Varnish lifting at joint in toe rail etc.

NO, it's getting wet because the joint is too tight and with heating and cooling cycles it expands and has nowhere to expand to and is lifting allowing water to get underneath......as shown by the black drip line in the lower right bung. I would've sliced the joint open more and then filled with the wood colored putty designed to fill joints. At this point in the game, I would try recaulking where it meets the fiberglass.

No idea. I've never used Rust-oleum varnish, but for what varnish costs I'd rather use a marine grade varnish such as interlux.

If it's not water-clear in the can; it will add color. Some varnishes are amber colored in the can, like Schooner Gold, and some are very dark, almost a black coffee color in the can, like Man O' War.
The only wood finish I have used that is clear after application is water-based. It looks like non-fat milk in the can, but dries clear.
Bar-top type epoxies may work, but be sure and determine the UV tolerance if it is outdoors, or even near a window.
You may be best off with an automotive clear coat.