If it were a poor connection anywhere in the system, including the battery, woulnd't it be the same regardless of whether the charger is hooked up or not?
Another consideration, take a battery known to be good, verify it is fully charged, then replace with the battery on the boat and retest. Though I don't think this test is necessary, as it will only confirm the previous battery is bad. I still think SS3964spd is right. Any pump will require far more amps than a simple electric fan. A hydraulic pump such as a trim pump in particular (more so even than a simple freshwater pressure pump). If it works with supplmental power but not on it's own, it's the battery.