Scalable weather database for wetter.at and wetter-deutschland.com

Reliable and precise weather forecasts are essential for planning private leisure activities as well as for companies and organizations. Mobile World Information Systems GmbH (MOWIS) offers weather services for this purpose and operates the websites wetter.at and wetter-deutschland.com, among others.

Development by RISC Software GmbH

RISC Software GmbH developed the central data management system for these services on behalf of MOWIS. Since 2011, it has been providing ongoing support and expanding it as required. At its heart is the NoSQL database HBase, which runs on a Hadoop cluster. Current weather data from various sources is continuously fed into it: the results of various forecast models as well as measured values from weather stations. On this basis, interactive weather forecasts can be called up worldwide up to 14 days in advance.

The system automatically selects the appropriate database for each forecast and combines information from multiple sources. It also temporarily stores data that has already been retrieved in order to further accelerate queries. An XML-based interface allows web-based services to retrieve data. In addition, RISC Software also provided SQL access with the help of Apache Hive. This is not used for export, but for interactive control of data quality by MOWIS.

Ongoing data import and export

The database in HBase is kept up to date through the ongoing import of forecast model results and measurement data from weather stations. The entire weather data supplied by a model is converted into a structured text representation, which subsequently enables the use of Hadoop MapReduce and HBase bulk imports. This means that a high-resolution weather forecast model for Austria can be imported over several hours within five minutes and thus made available for forecasts in real time. Similarly, a global weather data model with forecast values for one day can be imported in fifteen minutes. A comparable data import took several hours using the legacy SQL database that was replaced.

Design of a suitable data model

The data model was adapted to the queries so that the queries could be carried out interactively, enabling a quick query for a location, for example. In order to use a NoSQL database effectively, it is essential to design a data model that is optimized for the planned queries. For this reason, the planned queries were defined together with the domain experts from MOWIS at the start of the project. On this basis, the data model for HBase was defined, which in particular allows fast queries to individual locations and also enables the automated removal of data that is no longer required. In order to enable efficient access via other attributes, numerous lookup tables were also implemented. The use of a big data system enables flexible adaptation or expansion of the data model if new queries are required.

Acceleration and cost savings compared to the legacy SQL database

By switching to a Hadoop-based NoSQL solution, an additional forecasting model for global weather data was introduced and data imports and exports were accelerated by a factor of seven. This makes it possible to retrieve worldwide weather forecasts interactively or to export current weather forecasts for the whole of Austria and Germany for the above-mentioned websites. For this purpose, both the imports of the different weather models and the exports of the data updates for wetter.at use Hadoop Map-Reduce jobs to execute the creation of the current weather forecasts for the whole of Austria and Germany on the Hadoop cluster in parallel.

The use of Hadoop offers the following advantages:

Simple and cost-effective scaling of the system as data volumes grow by adding new computing nodes to the cluster.
The use of open source technology enables significant license costs to be saved compared to traditional commercial database offerings.