Make sure you do not have any routers EXCEPT for your backup server. It's typical that sysadmins configure the default gateway on every NIC: remove that and add a persistent route to your backup server through the interface you want to use for backups.
[since you didn't include the information necessary I can't be more precise than that]