Knowing what engine you have would be helpful but Iím going to assume itís a Mercruiser 3.0. I would not have thought the manifold as the issue since itís water jacketed and you didnít mention any water leaking. I would suspect the exhaust pipe to the inner transom plate. Although if you have an exhaust bellows instead of the tube and the bellows tears the exhaust will be louder.
1997 Silverton 362, 7.4 Crusaders
1997 2400 SCR, 5.7 Vortec / Bravo 2