Rehat 5.11 Fileserver dual nic,dual subnet config

I have two different switches that go to my sonicwall firewall, 

switch a is on the 10.0.1.x subnet and switch b is on the 10.0.2.x subnet.

On switch A is a Redhat 5.11 File server with static network IP 10.0.1.5 via eth2 which is a 10gbe etehrnet card.

right now the server  has 2 active network connections - 

IB0 is the infiniband connection to a separate subnet 192.168.100.x this network is local and doesn't goto the firewall, and has a bunch of servers and the other flames on it used for speedier file transfers for the flames.

eth2 which is a 10gbe nic and is currently connected to the switch on 10.0.1.x and gives me access to 

network, 10.0.1.5/255.255.255.0 gateway 10.0.1.1

I have 4 spare 1gb ports on the server

I need to be able to mount the server on the  10.0.2.x network  without going through the firewall, as the speed through the firewall is horribly slow due to packets being analyzed.

The long term and correct solution seems to be a switch inbetween the firewall and the subnets, however i need a quick fix right now to give a few clients access to the server without hitting the sonicwall.

So i was thinking about running a cat 6 cable from eth1 nic to switch 2 - 10.0.3.x network switch and assigning it an ip 10.0.3.5/255.255.255.0 gateway 10.0.3.1, but then on reading i am wondering wether i need to put the gateway in at all on this connection?

However i want to make sure that packets to and from that subnet will only go in and out over that nic and not get lost or try going through the firewall

if i look at the ip route on the server i currently see the following

192.168.100.0/24 dev ib0 proto kernel scope link src 192.168.100.19
10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.17
169.254.0.0/16 dev ib0 scope link
default via 10.0.1.1 dev eth2

i am concerned with trying this and freaking out the network causing broadcast storms etc, so i am looking to clarify what the correct way of doing this would be

thanks for any advice