Mastering the (industrial) Data (EN)

Merging Data: Big Data Architecture using the RISC Software Data Analytics Framework

The concept of the Big Data architecture of the RISC Data Analytics Framework is based on the combination of Big Data technologies with semantic approaches to process and store large amounts of data from different sources. The architecture is able to process data with Big Data technologies such as Spark. The data itself and the connections to each other are stored separately. This makes the system generic, reusable and yet responsive. The obligatory check of the data for correctness (data validation) is thus just as guaranteed as continuous analyses. Apache Spark is used – a parallel in-memory data processing framework that transfers the data from the raw format into a standardised storage – in order to carry out further analyses by data scientists with this database.

The processing of the data takes place in a cluster, into whose data storage the data, which originate from different data sources, are transferred. This transfer into a central memory includes an interpretation and transformation of the data in order to bring them into a uniform format. At the syntax level, the structure of this format is defined by reading in and processing the associated metadata information. In this way, the data is validated, cleaned, transformed accordingly and stored in Spark tables. By using Spark, one can perform initial data explorations directly on the cluster to gain first insights into the data. As a further step in data preparation, aggregation of data can also be carried out, enabling complex analysis tasks to be performed locally outside the BigData cluster. An example of this is the comparison of high-frequency time series described below.

In order to connect the relationships of the different data sources (i.e. represent Linked Data) and achieve a more powerful metadata representation, the Data Scientists of RISC Software GmbH rely on the schema language SALAD (Semantic Annotations for Linked Avro Data), which enriches Apache AVRO as a structural data storage format with semantic meta-information. This in turn enables an automated interpretation of the data. Furthermore, the pre-processing, structural validation and verification of linked documents is supported by a semantic set of rules. This closes the gap between record-oriented data modelling (supported by Apache Avro) and the Semantic Web (by extending SALAD).

Info

The video shows the architecture of the RISC Software Data Analytics Framework.

Prepare data: Spark

Data preparation is generally of great importance in this environment, as the data from customers and project partners is usually delivered in the raw form in which it was generated by the industrial machines. First of all, the various raw forms of the data have to be converted into a uniform format. This is the basis for further analyses. The reusability of the subsequent data analysis steps across a wide range of use cases is another important goal.

The data preparation as well as the subsequent analysis of the data is carried out using a Hadoop cluster, which on the one hand enables the efficient processing of data in the terabyte range, but on the other hand also specifies the target formats of the data preparation. The cluster enables data-parallel, horizontally scalable processing of data in the terabyte range, using open source software technologies and commodity hardware, which keeps the costs of deployment low and makes this solution very interesting, especially for SMEs.

The goal is to extract and summarise the relevant tables and files from the individual source databases to enable further efficient processing by the Hadoop framework. The result of the data preprocessing are CSV files that correspond to the desired schema and can be queried directly via Spark SQL after the one-time definition of a table schema. CSV-based tables can, for example, also be transferred to another storage format such as compressed and partitioned parquet tables with the help of Spark SQL. The use of SQL as a query language allows data scientists to easily aggregate and combine the data.

The video shows the use of the distributed processing framework Spark in a typical data engineering task, the conversion and preparation of industrial data.

Analysing data: Comparison of high-frequency time series

Time series are data that occur during continuous series of measurements, such as during the running time of a production machine. Comparing two machines that are identical in construction but do not work synchronously can lead to important conclusions. This also occurs in production; for example, two machines are started synchronously, but after only a few components have been produced, their cycle time is shifted. Using a method from the field of time series analysis called Dynamic Time Warping, it is possible to determine where exactly in the programme the lead/delay occurs as well as its magnitude. This information cannot be read from the comparison of the raw time series, especially if the behaviour between the time series keeps changing. A similarity matrix, in which two such time series are compared with each other, makes it possible to quickly gain an overview of all the time series under consideration and to recognise potentially problematic time series (errors, outliers). The value in the similarity matrix measures in a number the relevant distance between two time series for the respective use case.

The data considered in the video are high-frequency time series (sampling rate approx. 6ms), which hardly differ with the naked eye. With the help of Dynamic Time Warping (DTW), a common method for comparing curves, it was possible to determine where exactly in the programme the lead/delay occurs and its size. The similarity matrix shown at the end of the video makes it possible to quickly gain an overview of all the time series considered and to identify potentially problematic time series (errors, outliers), which are then subjected to a precise comparison in the next step using DTW path analysis (in video before).

Contact