Adding a new MVMe disk to my FreeBSD server

I’m running home servers since many many years (20+). In the past based on Debian. Then later based on FreeBSD because I liked the features of ZFS a lot. Having all the raid related commands under the hood makes it really easy to use.

I also have to run some Windows virtual machines. For this I’m using bhyve. The images are located on a raid1 pool of two spinning hard disks (WD red 4TB). It works, but it is not as fast as I would like to have it. Therefore my plan is to move the guests to a new MVMe drive.

Step one: Hardware

After some searching I selected a disk from WD which is recommened by the manufacturer for NAS usage. The server shows the following information. nvd0 is my root system. nvd1 the new MVMe drive.

nvd0: <CT1000P1SSD8> NVMe namespace
nvd0: 953869MB (1953525168 512 byte sectors)
nvd1: <WD Red SN700 2000GB> NVMe namespace
nvd1: 1907729MB (3907029168 512 byte sectors)

Step two: Preparing ZFS

The next step was then to create a partition and the pool. The FreeBSD manual is a good source of information: https://docs.freebsd.org/en/books/handbook/zfs/

sysctl vfs.zfs.min_auto_ashift=12
gpart create -s gpt nvd1
gpart add -t freebsd-zfs -l data nvd1
zpool create data gpt/data

Step three: Copy over data

Now set compression to on and start copying over the data from the spinning disks to the new pool. Finally create a snapshot and backup the snapshot via zfs send …

zfs set compression=on data
rsync -rav --info=progress2 /media/vm/* /data/vm/ 
zfs snapshot data/vm@20230803
zfs send -R data/vm@20230803 > /media/vm-bak/20230803

Step four: Testing the backup

mkdir /data/vm/test
zfs receive data/vm/test < /media/vm-bak/20230803
zfs destroy -r data/vm/test

Content was back in test – good.

Done!