Banker who used joint account to pay for affair stripped of millions in court fight