While investigating rsync/ssh performance, I discovered that our Solaris Clusters had just over half the expected throughput on their gigabit NICs.
This was caused by the cluster/streams related module clhbsndr being automatically pushed to each public NIC via /etc/iu.ap .
- Round trip time via ping was higher than non-cluster servers on the same network
- The throughput measured with iperf (and other tools) was around 400-600MBit/s instead of 940Mbit/s
- The clhbsndr module is loaded for the public interfaces, eg
ifconfig e1000g1 modlist 0 arp 1 ip 2 clhbsndr 3 e1000g
We have confirmed that bug 6625886 is the cause and are now waiting for Oracle to say if the module can safely be removed from the public interfaces of a Solaris 10 cluster, eg:
ifconfig e1000g1 modremove clhbsndr@2