Anyone help explain this Windlass Breaker?

Here is a link that will describe your breaker better. Then compare it to a Bussman equivalent, if one exists.

http://www.eaton.com/ecm/groups/public/@pub/@electrical/documents/content/br5401se0002a.pdf

I interpret the shut to go to a meter expressing current draw of the windless in either amps or percentage of full load current depending on the meter. "Since the shunt output is low voltage, light-gauge wiring can be used from shunt to meter. Indication may be displayed in percent, watts, safe/danger or other dial calibrations."
Do you have a % meter somewhere for the windlass or was the shunt ever even connected to anything ?

More importantly is the time delay and load carrying capability of the Eaton breaker.
I did not look at any of the Bussman line of breakers so I can't tell you how they compare but it looks to me like the Eaton breaker you have is designed for a motor load both for the inrush and the variable, heavy load that you would expect to pull a boat against the wind or current.

If the Bussman breaker is cheap enough, maybe you want to take a chance but the first mate will tire quickly running from the bow to the breaker to reset it.

I see the "C" termination is a spade connection on the bottom of the breaker. You said it was " a lighter gauge positive" connected there. Does that mean you tested the "C" wire and found 12 volts or does that mean it is just a red or any color other than black ?
This would be good to know if it had a constant 12 volts on it.

Sometimes a shunt is used to trip a breaker remotely. This would have nothing to do with a meter or the info in the link. Probably the breaker you have predates the info in the link.
Is it possible the windless has a safety limit, like on a hoist, that trips the breaker if you wind it up too tight ?

You need to trace the wires and find the other end.
I'd look at the windless termination first just in case the is a limit, if not, then you need to trace the cables. Easier said than done, I'm sure.
I think Home Cheapo sells a tracer that you can put a tone on the conductor and trace it that way too.
I have traced wires this way but with mixed results. If the cable is spliced somewhere weird, like above the head liner, to another wire or three, it can send you to the nut house.

Another thought, is the "C" terminal hot ( 12 volts ) when the breaker is connected to the # 1/0 cables.
In other words does the 1/0, when energized, make the "C" termination hot ?

Could this have anything to do with two sets of switches, on the bow and helm? Maybe protection from running the two sets in opposite direction? I assumed that would just trip my breaker, but not entirely sure.

Is the windlass breaker hot-wired to the battery? I don't think it would be under the main DC breaker, and it might need extra load protection. Is this an auto-reset breaker?

Shunt Trip

Some times it is advantageous to turn a breaker off from a remote location. To facilitate this task, an accessory called a shunt trip feature is installed by the manufacture inside of the breaker. This device consists of an electro-magnetic trip coil that is connected in series with an external field wired switch. When the switch contacts are closed, power is passed to the shunt trip coil causing the breaker’s mechanical latch to move to the open position. Re-closing the breaker is done by physically going to the breaker and manually moving the operating handle to the on-closed position.

One would need to look at a complete wiring diagram to verify the shunts intended purpose as there ae many reasons for a shunt to be used . Does your winch have a Emergency stop button ?

Last night I re-sealed my deck switches. One isn't working right, can't raise the anchor, new ones are on order. I tried the bridge windlass switches, which didn't work. I had the DC main breaker off, I turned it on, and the bridge switches worked. There must be a relay in there somewhere, maybe the windlass breaker also contains a relay.