The Lloyds banking group, which includes Halifax, Lloyds, Royal Bank of Scotland and TSB, first reported problems on Sunday afternoon.
Curran Annetts, a Lloyds bank customer from Hastings was on a date on Sunday afternoon and when he encountered the problem.
He said: “we went out, had a lovely meal and when we went to pay, my card got rejected. Tried her card and it was rejected as well cause we’re both with Lloyds and we went to try the cash machine but we still couldn’t get money out but the restaurant allowed us to pay the next day, which was a life saver.”
In a statement, Lloyds banking group said: “We apologise that earlier today (Sunday), between 3pm and 6 pm, some customers were unable to complete their debit card transactions.”
When the problems first arose, TSB chief Mr Pester tweeted: "My apologies to TSB customers having problems with their cards. I'm working hard with my team now to try to fix the problems."
He also took to Twitter to respond directly to unhappy customers.
He told the BBC that two of the seven servers that process TSB debit card payments had suffered problems, which meant that about a quarter of TSB card transactions, were affected.
Mr Pester later tweeted: "The cause of the problems at TSB is now fixed. It'll take a while to sort the backlog. Sorry customers may have problems for an hour or so."
Prof Alan Woodward, a cyber security specialist at the University of Surrey, told the BBC: "It is quite unusual for two bits of hardware to fall over at the same time so I suspect it's something else that's happened."
"Quiet times like Sundays are when they do things like maintenance, so they'll do software upgrades or they might be doing back-ups. And it's those sorts of things that tend to go wrong. So I suspect something like that has affected two machines at once."