API src

Found 5 results.

Other language confidence: 0.9888624932183305

DASF: A data analytics software framework for distributed environments

The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments. The data analytics software framework DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RCP) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RCP methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls. One application using the framework is the Digital Earth Flood Event Explorer (https://git.geomar.de/digital-earth/flood-event-explorer). The Digital Earth Flood Event Explorer integrates several exploratory data analysis tools and remote procedures deployed at various Helmholtz centers across Germany.

DASF: A data analytics software framework for distributed environments

The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments. The data analytics software framework DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls. One application using the framework is the Digital Earth Flood Event Explorer (https://git.geomar.de/digital-earth/flood-event-explorer). The Digital Earth Flood Event Explorer integrates several exploratory data analysis tools and remote procedures deployed at various Helmholtz centers across Germany.

DASF: Progress API: A progress reporting structure for the data analytics software framework

DASF: Progress API is part of the Data Analytics Software Framework (DASF, https://git.geomar.de/digital-earth/dasf), developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de). It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/). DASF: Progress API provides a light-weight tree-based structure to be sent via the DASF RCP messaging protocol. It's generic design supports deterministic as well as non-deterministic progress reports. While DASF: Messaging Python provides the necessary implementation to distribute the progress reports from the reporting backend modules, DASF: Web includes ready to use components to visualize the reported progress.

TERENO (Northeast), Soil moisture station Sassen BF1, Germany

The Sassen BF1 soil moisture station is part of an agrometeorological test site and aims at supplying environmental data for algorithm development in remote sensing and environmental modelling, with a focus on soil moisture and evapotranspiration.The site is intensively used for practical tests of remote sensing data integration in agricultural land management practices. First measurement infrastructure was installed by DLR in 1999 and instrumentation was intensified in 2011 and later as the site became part of the TERENO-NE observatory. The soil moisture station station Sassen BF1 was installed in 2012. It is located next to a pylon on a crest of an undulating field. The station is equipped with sensor for measuring the following variables: ScemeSpadeSoilMoisture_Spade_2_Temperature, ScemeSpadeSoilMoisture_Spade_6_Temperature, ScemeSpadeSoilMoisture_Spade_1, ScemeSpadeSoilMoisture_Spade_2, ScemeSpadeSoilMoisture_Spade_3, ScemeSpadeSoilMoisture_Spade_4, ScemeSpadeSoilMoisture_Spade_5 and ScemeSpadeSoilMoisture_Spade_6. The current version of this dataset is 1.5. This version includes two additional years of data (from-year to-year)and a revised version of the data flags. New authors were added for this new version: Alice Künzel (GFZ Potsdam), Christian Budach (GFZ Potsdam), Nils Brinckmann (GFZ Potsdam), Max Wegener (DLR Neustrelitz) and Klemens Schmidt (DLR Neustrelitz).A detailed overview on all changes is provided in the station description file. Older versions are available in the 'previous_versions' subfolder via the Data Download link. A first version of this data was provided under http://doi.org/ containing the measured data only. The dataset is also available through the TERENO Data Discovery Portal. The datafile will be extended once per year as more data is acquired at the stations and the metadatafile will be updated. New columns for new variables will be added as necessary. In case of changes in data processing, which will result in changes of historical data, an new Version of this dataset will be published using a new doi. New data will be added after a delay of several months to allow manual interference with the quality control process. During October 2020 a Bug in the published data was detected and a new version of the datasets was released from beginning until mid 2020. Data processing was done using DMRP version: 1.8.4. Metadataprocessing was done using DMETA version: 1.2.0.

DASF: Messaging Python: A python RCP wrapper for the data analytics software framework

DASF: Messaging Python is part of the Data Analytics Software Framework (DASF, https://git.geomar.de/digital-earth/dasf), developed at the GFZ German Research Centre for Geosciences. It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/). DASF: Messaging Python is a RCP (remote procedure call) wrapper library for the python programming language. As part of the data analytics software framework DASF, it implements the DASF RCP messaging protocol. This message broker based RCP implementation supports the integration of algorithms and methods implemented in python in a distributed environment. It utilizes pydantic (https://pydantic-docs.helpmanual.io/) for data and model validation using python type annotations. Currently the implementation relies on Apache Pulsar (https://pulsar.apache.org/) as a central message broker instance.

1