Storage#
Lustre#
This document grives a brief explanation on how I build Lustre on CentOS 7. Bear in mind, these are the steps taken for one specific installation.
I took the information for this page from Lustre Wiki and Whamcloud. I never understood which one is outdated and which is currently used, I got the feeling that both are both.
Contents#
Remarks before we start#
It is VERY important, that all kernel packages match! Yum tends to install wrong versions if they are not precicely specified!
I will start this document with the dependencies, but it might make sense to check, that your running kernel is supported by Lustre! So make sure to check the file lustre-release/lustre/Changelog for the tag you want to build, if it supports your kernel!
Check your current running kernel with
uname -r
Dependencies#
On a newly installed CentOS the security-aware troll inside me asks you to run
yum -y update
reboot
first. But remember, this might install a new kernel (that’s the reason for the reboot) and you have to remember lustre kernel support.
Then install
yum -y groupinstall "Development Tools"
If this does not work for any reasons, this is supposed to be a list of needed development packages:
yum -y install automake xmlto asciidoc elfutils-libelf-devel zlib-devel \
binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed \
rpm-build make gcc redhat-rpm-config patchutils git
Other dependencies are:
yum -y install xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel \
newt-devel python-devel hmaccalc perl-ExtUtils-Embed bison elfutils-devel \
audit-libs-devel libattr-devel libuuid-devel libblkid-devel libselinux-devel \
libudev-devel "kernel-devel-uname-r == $(uname -r)"
yum -y install epel-release
yum -y install pesign numactl-devel pciutils-devel ncurses-devel libselinux-devel \
fio openssl which libyaml-devel wget e2fsprogs vim openssl-devel keyutils
The "kernel-devel-uname-r == $(uname -r)" part was once very important, because like I said, yum tended to install non-matching kernel versions. This does not seem to work anymore. Maybe yum learned to match versions? This is important and must be checked!
So, make sure that kernel-devel is installed, even in the case the command above fails.
E2FSProgs#
This is a package needed for the tests (and maybe LdiskFS). To install it with yum (which is advised in the lustre documentation) you have to add it to the repos:
cat <<EOF > /etc/yum.repos.d/e2fsprogs.repo
[e2fsprogs-el7-x86_64]
name=e2fsprogs-el7-x86_64
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/
enabled=1
priority=1
EOF
and update it with yum update e2fsprogs.
Remark: This package might not be signed. This will result in an error while installing the LdiskFS packages. My solution was to install them with
yum install --nogpgcheck ...which is probably bad.
LdiskFS and ZFS#
Lustre supports two underlying filesystems for the OSDs: LdiskFS (which is their name for EXT4) and ZFS. One of these is enough to get a working Lustre, but the build will fail, if none of them can be installed. I leave it to you, which one you want to try.
LdiskFS#
For LdiskFS you need the EXT4 source code. This is part of the packages kernel-debuginfo and kernel-debuginfo-common-x86_64. These seem to be a bit hard to find.
What I did is
wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-KERNEL-VERSION.rpm
wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-KERNEL-VERSION.rpm
yum -y localinstall ./*.rpm
Make sure to replace KERNEL-VERSION with your current kernel.
Then you should be good to go.
Remark: This did not work with the newest master branch of lustre by the time of writing. I got compile errors in the utils with some EXT2 stuff and got rid of it by using the newest Release Candidate Tag/branch of lustre… Good luck!
ZFS#
I used to install ZFS by using the following commands:
yum -y install http://download.zfsonlinux.org/epel/zfs-release.el7_6.noarch.rpm
yum-config-manager --disable zfs
yum-config-manager --enable zfs-kmod
yum -y install spl zfs kmod-spl kmod-spl-devel kmod-zfs kmod-zfs-devel libzfs2-devel
Most recently, this resulted in a non-working zpool. I have no idea why, but please remember to update this document if you find a solution!
Lustre source#
Finally, lustre itself. Please check the Remarks before we start, so that you are warned if you are just using the newest version of the repo:
git clone git://git.whamcloud.com/fs/lustre-release.git
If you think you are all set up cross your fingers and run
cd lustre-release
sh autogen.sh
./configure
make rpms
Get a nice cup of coffee and when/if that all finished install it with
yum -y localinstall *.x86_64.rpm
Mounting and testing Lustre#
Since lustre is a parallel filesystem you will want to install clients and server versions on different systems and setting all this up… Have fun and maybe read the lustre-manual (a new version, there are ancient ones all over google)!
If you just want to check if it worked, the tests are a nice place to start. They are located in /usr/lib64/lustre/tests. You have a configuration in cfg/local.sh that will be used by default.
This config uses Ldiskfs, so if you only installed zfs run
truncate --size=2048M /tmp/lustre-{mdt1,ost1,ost2}
sed -i -e 's/FSTYPE:-ldiskfs/FSTYPE:-zfs/g' cfg/local.sh
The file llmount.sh will mount this local lustre filesystem in /mnt. To clean up all processes when unmounting, use llmountcleanup.sh
If you want to test if lustre is working as it should, you are supposed to run the tests with auster:
./auster -rv -d /opt/results runtests
The flags in this command mean:
flag |
meaning |
|---|---|
-r |
clean up, reload and rerun everything |
-v |
verbose output |
-d |
save output to file |
Lustre HSM#
The Hierarchical Storage Manager (HSM) of Lustre is a way to archive parts of Lustre. It can be configured in various ways with a lot of different options. Some information on this can be found in ICHEC’s LTFS Google drive folder, most of which is taken from the lustre manual again, as mentioned above.
After mounting Lustre (and adding at least one file to archive) HSM has to be enabled with this command:
lctl set_param mdt.lustre-MDT0000.hsm_control=enabled
When this is enabled, a copytool has to be started. This can either be the generic Lustre Posix copytool or any other copytool, like the one you are browing right now, if you found this file.
After starting the copytool (which is not in the scope of this document), files can be archived from Lustre using
lfs hsm_archive --archive=3 /mnt/lustre/file
The numer 3 in that command can be anything, but it must correspond to the copytool that should be used (usually those are started with a -A 3 option). You can set the default archive number with lctl set_param -P mdt.lustre-MDT0000.hsm.default_archive_id=3 as well.
The archived Lustre file can then be released (contents removed from lustre), restored (which also happens automatically when the file is accessed) and removed (deleted in the archive) with similar commands:
lfs hsm_archive FILE1 [FILE2...]
lfs hsm_release FILE1 [FILE2...]
lfs hsm_restore FILE1 [FILE2...]
lfs hsm_remove FILE1 [FILE2...]
What the current state of a file is can be checked with
lfs hsm_state FILE1 [FILE2...]