Edinburgh Data-Intensive Research - rapidInnovation

RapidSeis: Enabling User-Defined Seismological Waveform Data Processing over the Grid

Jano.van.Hemert — Mon, 03 May 2010 11:21:53 +0000

Speaker(s):

Presentation Type:

talk

The objective of this JISC-funded pilot project was to remove perceived barriers to uptake of an application that performs analysis of seismic waveform data. The aim was to provide the seismological community with a simplified system that overcame important barriers such as installation and understanding of the analysis package, location and transfer of large amounts of input data and visualisation of results.

The project combined the expertise of three distinct teams. The Orfeus Data Centre provided discovery services and access management to distributed earthquake waveform data and metadata, supported by tech- nologies such as webservices, SPARQL/RDF and a JSR-168 portal framework based on the WSRP portlet architecture. Analysis and visualisation of the waveform data were performed by SDX (Seismic Data eXplorer) developed at the University of Liverpool. SDX formed the core processing engine - this application’s functionality can be extended via user-defined algorithms coded as plugins. The processing engine was supported by several webservice-enabled modules used for data and plugin exchange. The user, data and processing infrastructure were connected using the Rapid framework created by the UK National e-Science Centre. Rapid generates intuitive interfaces to the processing core in the form of two self-contained JSR-168 remote portlets. One portlet facilitated plugin creation, whilst the other allowed execution of plugin code within the processing core running on a campus or national Grid infrastructure. The RapidSeis system was deployed within a community gateway: the NERIES web portal.

Although RapidSeis originated within the seismological community we suggest it forms a framework which could be exploited in different domains or scientific gateways where users can discover data, store data within a cart and wish to apply specialised processing algorithms remotely through a web browser. Processing performed on Grid infrastructure is completely transparent: the execution portlet handles submission and scheduling. The processing algorithms can be created online, stored and shared between users of the community web portal. The data and algorithm metadata are described in RDF and exchanged within the architecture which is commonly geographically distributed.
Further investigation is required to improve the provenance metadata to enable better traceability of both data and processing activities. The API used by the plugin code also requires additional development and the user-interface could be made even more intuitive with further modification of the Rapid framework.

Alessandro Spinuso (1), Andy Heath (2), Jos Koestjer (3), and Luca Trani (1)

(1) ORFEUS-KNMI, R&D Information and Observation Technology, de Bilt, Netherlands (spinuso@knmi.nl), (2) University of Liverpool, Department of Earth & Ocean Sciences, Liverpool England (aeh@liverpool.ac.uk), (3) e-Science Institute, Edinburgh, United Kingdom (jos@nesc.ac.uk)

Date and time:

Tuesday, 4 May, 2010 - 10:30

Location:

European Geosciences Union, General Assembly 2010, Vienna, Austria

Web site:

http://meetings.copernicus.org/egu2010/

Projects:

Rapid

Research topics:

Efficient distributed systems

Data-intensive computing

Collaborative environments

JISC:

Rapid featured in the newsletter of the Network of Research Infrastructures for European Seismology

Jano.van.Hemert — Thu, 04 Mar 2010 15:08:04 +0000

RapidSeis: a NERIES spin-off pilot

The objective of this six-month pilot project was to provide a simplified system to perform analysis of seismic waveform data through a web browser. The specific aims were that no data or application be download to the user’s computer, for the user to create algorithms to customise the analysis and to allow sharing of algorithms within the seismological community.

The project was funded by JISC who provided partial support for one of the researchers and allowed the three teams based at the University of Liverpool, ORFEUS and the UK National e-Science Centre in Edinburgh to meet regularly. The project culminated in a two day RapidSeis Workshop held at Liverpool in November 2009, where the results of the pilot project were presented along with representations from other groups about their latest developments in web services related to Seismology.

(source: http://www.neries-eu.org/ under "News and Events => Newsletter")

Topic of this submission:

JISC:

Research topics:

Projects:

Attachment	Size
NERIES Newsletter No8.pdf	755.1 KB

RapidSeis: Rapid Portals for Seismological Waveform Data

Jano.van.Hemert — Thu, 19 Nov 2009 16:10:22 +0000

Final project report for JISC, with links to all individually created deliverables and progress posts. RapidSeis has produced a scientific gateway via a web portal that allows seismologist to pick up data from Orfeus—the central repository for earthquake data in Europe—and then run several analyses on these data. Advanced users can also create new analyses and share these with all the other users.

When an analysis is run, data is extracted from the earthquake data stores in Europe and transferred to the University of Liverpool where the analysis is performed on a computer cluster. Once an analysis finishes, its results in the form of waveforms is send to the portal for the user to view and download.

* Screenshots or diagram of prototype: http://research.nesc.ac.uk/node/493
* Link to working prototype: http://193.52.21.80/jetspeed/portal/ needs login, so screencast provided next http://research.nesc.ac.uk/node/492
* Link to end user documentation: http://research.nesc.ac.uk/files/RapidTutorial-20090908.pdf
* Link to code repository or API: http://forge.nesc.ac.uk/projects/jos/
* Link to technical documentation: http://forge.nesc.ac.uk/download.php/455/Manual-0.9.2.pdf
* Date prototype was launched: 2009/11/13
* Project Team Names, Emails and Organisations: Jano I. van Hemert, j.vanhemert@ed.ac.uk - University of Edinburgh; Jos Koetsier, jkoetsie@staffmail.ed.ac.uk - University of Edinburgh; Andy Heath, aeh@liverpool.ac.uk - University of Liverpool; Alessandro Spinuso, Alessandro.Spinuso@knmi.nl - Royal Netherlands Meteorological Institute; Andreas Rietbrock, a.rietbrock@liverpool.ac.uk - University of Liverpool; Torild van Eck, torild.van.eck@knmi.nl - Royal Netherlands Meteorological Institute
* Project Website: http://research.nesc.ac.uk/rapid/
* PIMS entry: https://pims.jisc.ac.uk/projects/view/1340
* Table of Content for Project Posts
- RapidSeis workflow explained http://research.nesc.ac.uk/node/493
- Screencast: Rapid Portals for Seismological Waveform Data http://research.nesc.ac.uk/node/492
- Tutorial for Rapid http://research.nesc.ac.uk/node/487
- RapidSeis Workshop http://research.nesc.ac.uk/node/485
- Rapid portals improve teaching in practical sessions http://research.nesc.ac.uk/node/483
- Apache's Virtual File system is part of Rapid's architecture http://research.nesc.ac.uk/node/481
- Unit-testing code is especially important in code-generating projects http://research.nesc.ac.uk/node/482
- Apache's Maven for management of code structure? http://research.nesc.ac.uk/node/480
- Rapid prevents a disaster in the classroom http://research.nesc.ac.uk/node/479
- Rapid Tutorial at Advanced Distributed Services Summer School http://research.nesc.ac.uk/node/465
- Happy end user of Rapid gives praise http://research.nesc.ac.uk/node/442

Topic of this submission:

JISC:

Research topics:

Data-intensive computing

Collaborative environments

Intuitive interfaces

Projects:

Rapid

RapidSeis workflow explained

Jano.van.Hemert — Thu, 19 Nov 2009 16:02:16 +0000

RapidSeis consists of two porlets that run within the NERIES portal. Both assume the user logged in has already filled his/her cart with seismological data.

The Editor Portlet allows to (1) save and load source code of analysis plugins for the Seismological Waveform eXplorer application. When a load is performed, the code is loaded from a server that holds the personal account, which in turn asks (2) the description of the plugin, which is stored on the computing infrastructure. Whenever the user compiles (3) code, this builds a binary of the plugin, which is stored on the computing infrastructure (where it is needed when execution is requested). Plugins can be made public and thus shared with everyone that has access to the portal.

The Execution Portlet allows users to load (4) plugins from the account server. For each plugin the user interface is generated dynamically in the browser. Once parameters are set and the user requests (5) execution, the computing infrastructure gets the data in the user's cart (6) and runs the plugin on these data. The results are stored (7) on a web server in the form of HTML pages that contain aligned waveforms. These are then loaded by the portlet to allow the user to inspect and download them.

Topic of this submission:

JISC:

Projects:

Screencast: Rapid Portals for Seismological Waveform Data

Jano.van.Hemert — Thu, 19 Nov 2009 15:44:33 +0000

Welcome to the RapidSeis demo. This portal was created using Rapid and enables the use of Seismic Data eXplorer within the web portal of the Network of Research Infrastructures for European Seismology.

Topic of this submission:

JISC:

Research topics:

Projects:

Tutorial for Rapid

Jano.van.Hemert — Wed, 18 Nov 2009 15:28:41 +0000

This brief tutorial covers the installation of Rapid and the components it depends on. It explains how to get a simple portlet running and then increases the complexity of this portlet to include more functionality by exploiting Rapid's features.

Topic of this submission:

JISC:

Projects:

Attachment	Size
RapidTutorial-20090908.pdf	290.14 KB

RapidSeis Workshop

Jano.van.Hemert — Sat, 31 Oct 2009 11:12:05 +0000

At the meeting we will present the result of the RapidSeis project, a collaboration between ORFEUS, the UK National e-Science Centre and the University of Liverpool. Over the past six months, this project has created a system that facilitates running waveform analysis on data from ORFEUS where the computation is performed on remote compute resources provided by the University of Liverpool. The system is used via a secure web portal, which means no software needs to be installed by end-users; they only need access to a common web browser. Even visualization will be embedded in the portal.

The portal enables two modes of operation. First, it allows registered users to choose an analysis from a catalogue of analyses and then run this analysis on data chosen from the data hosted by ORFEUS. Second, it enables more advanced users to create new analyses and add these to the catalogue for others to use.

November 12th
• 13:00 Welcome/Lunch in the staff common room, Department of Earth and Ocean Sciences.
• 14:00 - 14:30 Luca Trani
• 14:30 - 15:00 Robert Barsch
• 15:00 - 15:30 Marco Olivieri
15:30 - 16:00 Coffee break
• 16:00 -16:30 Jano van Hemert
• 16:30 - 17:00 Paul Denton
18:30 Meet in the Philharmonic Pub for drinks, followed by dinner at 20:00 at Gusto Restaurant at the Albert Dock: http://www.gustorestaurants.uk.com/restaurants/gusto-liverpool

November 13th
• 09:00 - 10:45 RapidSeis - Andy Heath, Jos Koetsier, Luca Trani
10:45 - 11.15 Coffee break
• 11:15 - 12:30 RapidSeis
12.30 Lunch
• 14:00 Wrap up
15:00 Finish

Attendees (from left to right in the picture): Moritz Beyreuther (LMU), Robert Barsch (LMU), Paul Denton (BGS), Jano van Hemert (NeSC), Andreas Rietbrock (Liverpool), Andy Heath (Liverpool),Marco Olivieri (ETHZ), Alessandro Spinuso (Orfeus - facsimile), Luca Trani (Orfeus), Alexandra Lodge (Liverpool), Jos Koetsier (NeSC), Linus Kamb (EMSC)

Location:

Department of Earth & Ocean Sciences, University of Liverpool, UK

Dates:

12 Nov 2009 to 13 Nov 2009

Web site:

http://pcwww.liv.ac.uk/~aeh/Meetings/rapidseisws.htm

Research topics:

Data-intensive computing

Projects:

JISC:

Rapid portals improve teaching in practical sessions

Jano.van.Hemert — Thu, 29 Oct 2009 11:28:59 +0000

Written by Nick Funnel (2nd year PhD in the School of Chemistry).

The lab has been much easier to teach this year.

Last year the students had to edit Z-matrices for geometry optimization calculations, which have to be located from separate files and navigate between a PC and a workstation. This also required them to deal with UNIX in order to complete the lab. The vast majority of students struggled, having never dealt with issues like this before, and a significant proportion of these students were unhappy just using computers in general.

As a result, most of the calculations that were submitted did not work—most likely due to an input error but these were difficult to spot. Eventually, the answers often had to be provided to the student from a calculation that been previously run. Sorting out these problems was very time consuming and so there was no time to effectively teach the students the chemical theory behind the experiment and most students saw it as a waste of time.

This year, most of these problems have been solved—students no longer have to manage lots of separate files or use UNIX. Although the students still have to edit Z-matrices, there are clear instructions accompanying them and there is more time to explain the format of them to the student.

So far there has been no problem running any of the calculations. Students are working from one webpage and by effectively ‘clicking next’ all the time they are given more time to think about the theory behind what they are doing rather than just trying to get the calculations to run.

Topic of this submission:

JISC:

Projects:

Apache's Virtual File system is part of Rapid's architecture

Jano.van.Hemert — Mon, 19 Oct 2009 15:21:00 +0000

Apache's Virtual File system helped us to access different file systems transparently. The Rapid project implemented a new 'GSIFTP' (Grid Security Infrastructure File Transfer Protocol) module, which adds Grid connectivity to the set of supported file systems. This sub-project of Rapid, spawned the commons-vfs-grid module (http://sourceforge.net/projects/commonsvfsgrid/) which now supports GSIFTP, SRB (Storage Resource Broker) and iRods (https://www.irods.org).

Topic of this submission:

JISC:

Projects:

Unit-testing code is especially important in code-generating projects

Jano.van.Hemert — Mon, 19 Oct 2009 15:20:21 +0000

The use of testing frameworks, such as Junit is extremely valuable. It takes quite a bit of discipline to write new unit tests for each new piece of 'real' code that is written, but the benefits in the end in terms of stability, confidence and debugging make the extra overhead more than worth it. This is especially useful in a project such as Rapid, where code is being generated. Testing in the 'traditional' way, by examining or running generated code is extremely difficult and by using testing frameworks we can actually determine correctness of Rapid, before running any generated code.

Topic of this submission:

JISC:

Projects: