ZFS HOWTO
Introduction
The Zed File System (ZFS) is an advanced filesystem that merges a file system, logical volume manager, and software RAID. It consists of the following components:
- A ZFS virtual device (vdev) is logical device in a zpool, which can be a physical device, a file, or a collection of devices.
- A ZFS zpool is a logical group of devices describing the layout and physical characteristics of the available storage. Disk space for datasets is allocated from a zpool. RAID is implemented in a zpool and can be striped, mirrored, or raid-z.
- A ZFS dataset behaves like other file systems and is mounted within the standard system namespace.
- A ZFS snapshot is a read-only copy of a dataset at a given point in time.
Basic Commands
List zpools
# zpool list
Check zpool status
# zpool status
Create a zpool
# zpool create zpool -o ashift=12
Delete a zpool
# zpool destroy zpool
List datasets
# zfs list
Create a dataset
# zfs create dataset -o atime=off -o mountpoint=mountpoint
Delete a dataset
# zfs destroy dataset
List snapshots
# zfs list -t snapshot
Check to see how much space snapshots use
Note: Be very careful with this command! Snapshots may be deleted without the ‘-n’ (dry-run) argument. Snapshots that are deleted before they are replicated to a backup server cannot be recovered.
For example, the see how much space can be recovered by deleting snapshots that are older than one second (all of them):
$ /sb/admin/scripts/zfs-prune-snapshots -v -n 1s -p tank/hd1 removing tank/hd1@zfs-auto-snap_monthly-2023-02-01-00h28: 2 weeks old <dry-run: no action taken> would reclaim 1.89G removing tank/hd1@zfs-auto-snap_daily-2023-02-10-00h07: 6 days old <dry-run: no action taken> would reclaim 684M : removing tank/hd1@zfs-auto-snap_frequent-2023-02-16-15h45: 1 minute old <dry-run: no action taken> would reclaim 6.67M
View Snapshots
ZFS creates a special .zfs/snapshot directory which shows the snapshots as directories of the dataset as it existed at a particular time.
rinerwc@halibut /home/rinerwc% cd .zfs/snapshot rinerwc@halibut /home/rinerwc/.zfs/snapshot% ls zfs-auto-snap_daily-2023-02-10-00h07/ zfs-auto-snap_hourly-2023-02-16-00h00/ zfs-auto-snap_daily-2023-02-11-00h07/ zfs-auto-snap_hourly-2023-02-16-01h00/ zfs-auto-snap_daily-2023-02-12-00h07/ zfs-auto-snap_hourly-2023-02-16-02h00/ zfs-auto-snap_daily-2023-02-13-00h07/ zfs-auto-snap_hourly-2023-02-16-03h00/ zfs-auto-snap_daily-2023-02-14-00h07/ zfs-auto-snap_hourly-2023-02-16-04h00/ zfs-auto-snap_daily-2023-02-15-00h07/ zfs-auto-snap_hourly-2023-02-16-05h00/ zfs-auto-snap_daily-2023-02-16-00h07/ zfs-auto-snap_hourly-2023-02-16-06h00/ zfs-uto-snap_frequent-2023-02-16-14h45/ zfs-auto-snap_hourly-2023-02-16-07h00/ zfs-auto-snap_frequent-2023-02-16-15h15/ zfs-auto-snap_hourly-2023-02-16-08h00/ zfs-auto-snap_frequent-2023-02-16-15h30/ zfs-auto-snap_hourly-2023-02-16-09h00/ zfs-auto-snap_frequent-2023-02-16-15h45/ zfs-auto-snap_hourly-2023-02-16-10h00/ zfs-auto-snap_hourly-2023-02-15-16h00/ zfs-auto-snap_hourly-2023-02-16-11h00/ zfs-auto-snap_hourly-2023-02-15-17h00/ zfs-auto-snap_hourly-2023-02-16-12h00/ zfs-auto-snap_hourly-2023-02-15-18h00/ zfs-auto-snap_hourly-2023-02-16-13h00/ zfs-auto-snap_hourly-2023-02-15-19h00/ zfs-auto-snap_hourly-2023-02-16-14h00/ zfs-auto-snap_hourly-2023-02-15-20h00/ zfs-auto-snap_hourly-2023-02-16-15h00/ zfs-auto-snap_hourly-2023-02-15-21h00/ zfs-auto-snap_monthly-2023-02-01-00h28/ zfs-auto-snap_hourly-2023-02-15-22h00/ zfs-auto-snap_weekly-2023-02-12-00h14/ zfs-auto-snap_hourly-2023-02-15-23h00/
Backups
ZFS uses snapshot replication to backup datasets.
Snapshots operate on a block level. Once a snapshot is created, dataset blocks which change are copied to the snapshot (copy on write or COW). The original block is not changed. Snapshot initially are zero size and grow as the associated dataset changes.
Root cron jobs create snapshots frequently (every 15 minutes), hourly, daily, weekly, and monthly. Only a certain number of snapshots are kept: 4 frequent, 24 hourly, 7 daily, 1 weekly, and 1 monthly.
[root@trigger ~]# crontab -l #Ansible: Frequent snapshots 15,30,45 * * * * /bin/bash -l -c '/usr/local/bin/zfs-auto-snapshot frequent 4' #Ansible: Hourly snapshots 0 * * * * /bin/bash -l -c '/usr/local/bin/zfs-auto-snapshot hourly 24' #Ansible: Daily snapshots 7 0 * * * /bin/bash -l -c '/usr/local/bin/zfs-auto-snapshot daily 7' #Ansible: Weekly snapshots 14 0 * * 7 /bin/bash -l -c '/usr/local/bin/zfs-auto-snapshot weekly 1' #Ansible: Monthly snapshots 28 0 1 * * /bin/bash -l -c '/usr/local/bin/zfs-auto-snapshot monthly 1'
The daily, weekly, and monthly snapshots are replicated (copied) to a backup server each night and are kept for a maximum of 30 days. Using snapshots, only the changed blocks are replicated. This make ZFS replication much more efficient than rsync. Rsync must find the files that have changed (which can take a significant amount of time) and then copy each modified file to the backup server.
Best Practices
Keep ZFS files systems below 80% of capacity. Performance degrades quickly as the usage increases beyond 80%.
Miscellaneous
Using the ‘df’ command with ZFS datasets is confusing. It is not aware of ZFS snapshots, so as the amount of space used by snapshots increases, it sees the disk size decrease.