I doubt your bilge pump is wired directly to the battery. I suspect you need the Battery Isolation switch turned to ON for the bilge pump to work. There should be a common ground and 2 positive leads to the pump. One lead comes from the helm switch the other from the fuse bus.
If your helm switches (particularly the bilge pump switch) is a Lighted Switch, then when the float switch actuates the pump, the light on the bilge pump switch will turn on even if the switch itself is not on.
To address your question about being underway. You should be scanning your gauges (Water Temp.; Oil Pressure; Voltage, RPM, speed) along with the bilge switches every minute or so while underway. If you're not, I would strongly urge you to get into that habit. Audible alarms are for those who don't watch their gauges. If monitoring properly, then an audible alarm should should not be a surprise.