Previous Projects - Public#
Horizon Europe#
IO-SEA#
IO-SEA was an EU Horizon funded project undertaken as part of the Performance Activity, from 2021 to 2024. The overall project focused on the development of next-generation I/O and storage solutions for Exascale HPC systems.
ICHEC focused on the Hierarchical Storage Management (HSM) element, developing the Hestia software to support access to the long term storage backend of the IO-SEA software stack. The Hestia software built on Deimos which was created for the previous ICHEC ExSeisDat project.
Estuary#
Estuary is a Lustre S3 Copytool. It was forked from ComputeCanada’s lustre-s3-copytool with some minor updates. It was intended to use it as an element of the IO-SEA software stack, but an alternative method for data migrations was adopted in the end.
SFI#
ExSeisDat#
ExSeisDat was a LERO / SFI “spoke” project with Tullow Oil and Data-Direct Networks (DDN) as industry partners. It was undertaken as part of the Performance Activity from 2016 to 2020.
It lead to the development of the ExSeisDat Software for high-performance parallel I/O of seismic data files.
It also had several sub-projects, notably, Deimos and the GPUDirect investigations.
A number of smaller projects started off the back of this as well, including PERForate and Megadep.
The project came about as a result of consultancy work with Tullow Oil. It was found that the I/O of SEG-Y files was a significant performance bottleneck unless carefully handled. Parsing SEG-Y files is already a pain, and reading them in parallel is even worse.
Deimos#
Deimos was a project collaborating with CEA to integrate their Phobos storage with Lustre HSM using a webserver and the Amazon S3 REST API.
It used Proxygen to create a webserver that accepts HTTP requests that are conform to the AWS S3 REST API and saves/retrieves the data sent/requested to/from Phobos as a backend by using FIFOs.
Deimos’s functionality was later eventually subsumed into ICHEC’s Hestia sofware as part of continued collaboration with CEA in the IO-SEA project.
PERForate#
PERForate is a tiny performance measurement library. The idea is to include one header, and drop PERFORATE_SCOPE_TRACE("my text") into a function, and have some measurement of its execution time printed after the program exits, no extra magic, flags, or incantations.
This gets around a lot of issues wrt compiler optimizations inlining some functions, omitting symbols, finding the right incantation for the compiler on your system. Each call to PERFORATE_SCOPE_TRACE has about 100 ns overhead, so it can mostly be dropped in and ignored anywhere except for hot-loops.
This came about as a result of frustration with trying to use the “proper” tools to track down a performance bug in Megadep, when quite a simple measurement was needed.
Megadep#
Megadep is for determining the transitive dependencies of source files.
It aims to optimize reads from disk by determining the dependencies for all project files at once, necessitating a single pass through the source files.
One goal of this project was to develop a clean example project for the Performance Engineering group, presenting nice approaches to project structure and testing.
An ideal outcome would be where it can be converted into a cookie-cutter template for any future C++ projects in ICHEC.