PowerTSM®️ Time Series Management – Achieving Data Warehouse Quality in the Azure Cloud and Enrolling AI.

When we talk about the data world of the energy transition, we are talking about big data. That alone would be enough of a challenge to keep supply grids stable. However, the challenges that energy service providers already have to face today in terms of data technology alone, are much more demanding than just dealing with large amounts of data. An accelerating growth in data volumes, which goes hand in hand with increasing complexity, can be observed in all areas.

Parallel to this general phenomenon, the way in which data is processed, stored and distributed is changing. Edge devices take over the basic processing of data, store it and forward it to gateways. From there, compressed data sets reach cloud solutions. Traditional IT approaches would not even come close to being able to map this complexity or to deliver the required process speed.

The complexity prevailing at the IoT and gateway level continues at higher levels. Data finds its way into different cloud solutions, such as OCI, AWS or Azure, for example, also in on-premis solutions, and is stored in different formats in a wide variety of database systems, including event-oriented ones. In the end, if important decisions are to be made on the basis of this data, a fundamental question arises: What data quality are we dealing with? Is the data authentic, has it been manipulated, is it plausible and is it complete? It's all about trust, because decisions made on the basis of this data can have far-reaching consequences.

Let's take a look at how elegantly and seamlessly PowerTSM^® solves the issue of time series management and forecasting in this demanding environment: In the multi-cloud environment, in hybrid solutions from cloud and on-premis, polyglot in the database area and in terms of data quality. Azure Services help with simple data integration and setting up AI solutions, making PowerTSM^® a real holistic approach.

The infrastructure of a modern energy industry is highly networked. IoT, edge devices, gateways and 5G networks dominate the landscape. Computing power and storage have been moved to the edges of the network. They are a particular gateway for data theft or manipulation or attacks on the infrastructure in general. Securing local networks with effective firewalls is only one aspect of data security, because when data from LAN or WAN structures makes its way to the cloud or to on-premise solutions, it is fair game for hackers. Only end-to-end encryption provides security. To support Azure Cloud users with secure data traffic, Microsoft has launched the Azure Edge Secured Core program. As part of this program, devices that have to meet high requirements are certified: Encryption directly on the chip, update capability and the regular provision of security patches or the sealed delivery of the devices to the end customer are just some of the criterias.

PowerTSM^® fits seamlessly into the system of end-to-end encryption. Encrypted data that reaches the Azure Cloud is only decrypted again in the PowerTSM^® user's entity. PowerTSM^® makes decrypted data directly available in a corresponding time series and provides it with an authenticated flag, all others with an opposite flag. As flags are inherited in an upwardly scaled manner, it is possible to recognize whether the entire time series has been fully authenticated, even when selecting a rough time grid. This ensures maximum data security in the cloud without the slightest additional effort for the PowerTSM^® customer.

Verifying data is just one step, but a very important one, to ensure data quality, especially in the cloud. Another major factor is the question: Is the data presented actually plausible? A distinction must be made here between outliers, correct but unusual data, and data that is actually incorrect. The latter can result from measurement errors due to technical problems and much more. In any case, incorrect data must be eliminated from time series. This is not the case with outliers, as these can be an important source of information in predictive maintenance, for example.

PowerTSM® offers a wide range of options for checking data for plausibility. From the very simple, such as defining bandwidths within which certain data should move, to more complex procedures such as mirroring forecast data. Customers can use a toolbox or develop their own procedures based on Python or others. If data is identified that is not plausible, actions must be triggered to replace it if necessary. This brings us to the third topic in the area of data quality: Closing the Gap.

Gaps, missing data in time series, can be caused by technical problems. As a rule, they are the result of the
synchronization of a time series set with different time grids. This topic will become increasingly important in the future. Energy infrastructure is being covered with speed by IoT devices that constantly feed information into cloud solutions in non-standardized time grids, even from different time zones. In order to achieve data quality, for example to calculate accurate forecasts on the basis of the available data, data gaps must be closed in a meaningful way. It is therefore worth taking a look at what PowerTSM^® can do in this area to provide data of the highest possible quality.

PowerTSM^® provides a range of on-board tools to automatically close data gaps using routines. This starts with simple methods, such as using the average of n elements before and after the data gap as a placeholder, through to more complex methods. Forecast calculations, reference time series or the best picture method can be used to close data gaps. However, there are no limits for customers. They can implement their own developments using Python or use Azure AutoML to train data models for closing gaps.

Whichever solution or solutions PowerTSM^® customers choose to close data gaps, PowerTSM^® automatically logs any interventions in the original time series in the audit axis. This makes it possible to track at any time which data has been manipulated and on what basis. All the procedures mentioned here can also be used to replace non-validated data in routines. There is no need to mention that PowerTSM® can automatically solve the time zone problem. In addition to the audit axis, this is one of the many features that make PowerTSM^® so unique and an industry standard for the energy industry.

Our energy world is becoming increasingly complex, but with the right systems and tools, it won't get any more complicated. It is therefore time to question previous approaches and think in new ways. The future lies in the cloud and AI will help us to successfully meet the new challenges. PowerTSM^® and Azure are your partners on this journey. Learn more about PowerTSM®.