ISSUE: In a cluster environment, all VERDE servers (nodes) are sharing the same storage. If not sized properly (IOPS, bandwidth) the storage device can quickly become a bottleneck for the entire system and induce performance issues. This article describes how to quickly test the transfer rate between the VERDE server and the storage device to identify a possible I/O bottleneck.
Note that this information is not specific to VERDE and should be familiar to Linux administrators, additional information about the "dd" command is available from the Linux command reference.
SOLUTION: The command below (dd) does a block copy between a source and a destination location. You can first do a test copy to a local direct attached drive and then test the transfer rate to the remote storage, you will notice that the output is significantly different (it of course depends on your setup)
Copy to a local drive:
This example copies 10,000 times a 10KB file, for a total of 105MB.
dd if=/dev/zero of=/tmp/test.img bs=10k count=10k
It will provide an output similar to this, note that the transfer rate (388MB/s in this example) will defer depending on the specifications of your drives.
104857600 bytes (105 MB) copied, 0.0270529 s, 388 MB/s
Copy to a location on the shared storage:
On a VERDE cluster, the /home directory is located on the shared storage and contains the data shared by all servers.
dd if=/dev/zero of=/home/test.img bs=10k count=10k
This will return something like:
104857600 bytes (105 MB) copied, 1.26297 s, 83.0 MB/s
In this example the transfer rate is 83MB/s, as expected it is significantly lower that what was achieved on the local drive. But the question is: Is it good or bad?
On a Gigabit Ethernet network, this can be considered a good transfer rate. Do not forget that there may be traffic other than VERDE on the network, and in this example we are almost using the entire bandwidth. You will have to correlate this result with how many users are concurrently accessing the storage to evaluate what is available per user and verify if it is sufficient.
This gives you a quick indication on the quality of the transfer, for example, if the result is a low number 10, 20MBps... On a Gigabit network you will need to investigate to isolate what his accessing the storage or heavily using the network at the same time and is impacting the performance of the VERDE system.
Repeat this process on each server where you suspect storage issue.
If additional troubleshooting is required you will need to involve the Linux administrator and install system monitoring tools, like Sysstat for example.