He says negligent share advice from one of its advisers led him to lose £1.6m.
Mr Fisher, who played a major part in rescuing Town as part of the Ken Davy-led consortium, had hired Barclays Stockbrokers to manage a share portfolio for him at the height of the dotcom boom in 2000.
But he said bad advice from the broker assigned to him led him to lose a fortune in 20 months.
Mr Fisher said he lost more than £900,000 buying shares in software firm Izodia, formerly Infobank.
He said he bought shares in the business 10 separate times.
Mr Fisher says his broker continually advised him to carry on investing - despite the firm's tumbling share price - in the hope the price would eventually rise.
The company collapsed last December and was investigated by the Serious Fraud Squad.
Mr Fisher said he also made six separate investments totalling almost £175,000 in media firm Just Group, which went into administration in January last year.
In a letter to Mr Fisher in December, 2001, the broker who handled his share dealings admitted he had performed poorly and that his service had been "erratic".
But he blamed personal problems for his poor performance.
He also said that in the case of Just Group there had been no signs of trouble before trading in its shares was suspended.
Mr Fisher said: "A lot of people have lost money on shares over the past few years.
"If it had been as simple as that I would not be complaining - but this is different.
"This guy has admitted he gave a bad service and an erratic service and was not on top of his job."
Mr Fisher, who made millions with the sale of his Travelworld travel agency chain to Airtours in 1998, said he was a successful businessman.
Mr Fisher, who now has business interests in Spain, said he had been pursuing his case for over a year.
He added: "It is a very expensive process, but our legal advice is that we have a very strong case."
A spokeswoman for Barclays Stockbrokers said: "We cannot say anything about this case because it is with our lawyers and for reasons of client confidentiality."
But she added that the final decision to buy and sell stock rested with the client.