ZFS on Linux performance tuning when zfs_iput_taskq take near 100% CPU

# cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=10240000000

# zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
zp1          548M   267G   136K  /zp1
zp1/data01   302M   267G   302M  /data01
zp1/pg_log   980K   267G   980K  /pg_log
zp1/site     234M   267G   234M  /site
# zfs set xattr=sa zp1
# zfs set xattr=sa zp1/data01
# zfs set xattr=sa zp1/pg_log
# zfs set xattr=sa zp1/site

# vi /etc/sysctl.conf
vm.swappiness = 10
vm.min_free_kbytes = 512000

reboot

ZFS on Linux needs a bit of basic tuning in order to operate well under load. There's a bit of a struggle between the ZFS ARC and the Linux virtual memory subsystem.

For your CentOS systems, try the following:

Create an /etc/modprobe.d/zfs.conf configuration file. This is read during the module load/boot.

Add something like:

options zfs zfs_arc_max=40000000000
options zfs zfs_vdev_max_pending=24
Where zfs_arc_max is roughly 40% of your RAM in bytes (Edit: try zfs_arc_max=1200000000). The compiled-in default for zfs_vdev_max_pending is 8 or 10, depending on version. The value should be high (48) for SSD or low-latency drives. Maybe 12-24 for SAS. Otherwise, leave at default.

You'll want to also have some floor values in /etc/sysctl.conf

vm.swappiness = 10
vm.min_free_kbytes = 512000
Finally, with CentOS, you may want to install tuned and tuned-utils and set your profile to virtual-guest with tuned-adm profile virtual-guest.

Try these and see if the problem persists.

Edit:

Run zfs set xattr=sa storage. Here's why. You may have to wipe the volumes and start again (I'd recommend).