A no-brainer way to verify your IB network is working with MPI is to run an MPI benchmark and see if it is running at an IB speed.
The Administrator Manual contains instructions on how to run a PingPong test between two nodes. Do that and check the bandwidth (Mbytes/sec column) for large messages (1048576 bytes) and the latency (t[usec] column) for 0 byte messages.
If you are running over 1 GigE, you will see ~20-30usec latency and max 125MBytes/sec. If you are seeing lower latency and higher bandwidth numbers, you know for sure that your IB network is being used.