Development by RISC Software GmbH<\/h2>\n\n\n\n
RISC Software GmbH developed the central data management system for these services on behalf of MOWIS. Since 2011, it has been providing ongoing support and expanding it as required. At its heart is the NoSQL database HBase<\/a>, which runs on a Hadoop cluster<\/a>. Current weather data from various sources is continuously fed into it: the results of various forecast models as well as measured values from weather stations. On this basis, interactive weather forecasts can be called up worldwide up to 14 days in advance. <\/p>\n\n\n\n The system automatically selects the appropriate database for each forecast and combines information from multiple sources. It also temporarily stores data that has already been retrieved in order to further accelerate queries. An XML-based interface allows web-based services to retrieve data. In addition, RISC Software also provided SQL access with the help of Apache Hive. This is not used for export, but for interactive control of data quality by MOWIS. <\/p>\n<\/div>\n\n\n\n The database in HBase is kept up to date through the ongoing import of forecast model results and measurement data from weather stations. The entire weather data supplied by a model is converted into a structured text representation, which subsequently enables the use of Hadoop MapReduce and HBase bulk imports. This means that a high-resolution weather forecast model for Austria can be imported over several hours within five minutes and thus made available for forecasts in real time. Similarly, a global weather data model with forecast values for one day can be imported in fifteen minutes. A comparable data import took several hours using the legacy SQL database that was replaced. <\/p>\n\n The data model was adapted to the queries so that the queries could be carried out interactively, enabling a quick query for a location, for example. In order to use a NoSQL database effectively, it is essential to design a data model that is optimized for the planned queries. For this reason, the planned queries were defined together with the domain experts from MOWIS at the start of the project. On this basis, the data model for HBase was defined, which in particular allows fast queries to individual locations and also enables the automated removal of data that is no longer required. In order to enable efficient access via other attributes, numerous lookup tables were also implemented. The use of a big data system enables flexible adaptation or expansion of the data model if new queries are required. <\/p>\n\n By switching to a Hadoop-based NoSQL solution, an additional forecasting model for global weather data was introduced and data imports and exports were accelerated by a factor of seven. This makes it possible to retrieve worldwide weather forecasts interactively or to export current weather forecasts for the whole of Austria and Germany for the above-mentioned websites. For this purpose, both the imports of the different weather models and the exports of the data updates for wetter.at use Hadoop Map-Reduce jobs to execute the creation of the current weather forecasts for the whole of Austria and Germany on the Hadoop cluster in parallel. <\/p>\n\n The use of Hadoop offers the following advantages:<\/p>\n\n <\/p> ![\"\"](\"https:\/\/www.risc-software.at\/app\/uploads\/2023\/10\/iStock_foto-deckblatt-1024x683.jpg\")
<\/figure>\n<\/div>\n<\/div>\n\nOngoing data import and export<\/h3>\n\n
Design of a suitable data model<\/h3>\n\n
Acceleration and cost savings compared to the legacy SQL database<\/h3>\n\n
\n
Project partners<\/h2>\n\n\n\n
![\"\"](\"https:\/\/www.risc-software.at\/app\/uploads\/2023\/10\/MOWIS_Logo_WORLD_rgb20140305-31127-31mehl.jpg\"){kind=logo}
<\/figure>\n<\/div>\n\n\n\nProject details<\/h2>\n\n\n\n
\n
\n
Contact us<\/h2>\n\n\n\n
<\/ul><\/div>\n