Monday, May 9, 2011
The Systems Disaster
The US economy must be picking up. I received some email from US recruiters. There must be a demand for infrastructure and disaster recovery experts there. This is a coincident because I’ve been following up on a secondary disaster following the earthquake and tsunami. The Mizuho Bank’s systems disaster showed up a few days after 3/11.
Mizuho has reported that deposits into two earthquake-relief funds inundated their systems after March 12th but couldn’t handle it. “It” is the problem. We don’t know exactly what it is. The systems started limping afterward, but partially operating systems meant not working at all. The batch window is limited. It must complete by morning. I think the bank couldn’t balance all the accounts before the start of the following day.
This was in the middle of the Disaster. Affected people needed cash. The bank closed all ATMs for three days to recover, but they announced that the users could go to their branch offices and withdraw up to 100000 yen which is about $1200. Guess what? Some users went to their different branch offices and kept withdrawing 100000 yen even though they didn’t have money in their account. Talking of chaos! The problem didn’t come to end until 10 days later.
To compete in the global financial world, Mizuho was created by merging three large banks, Fuji, Daiich-Kangyo, and Kougyo banks. Not two but three banks at the same time. It kept the old accounting systems of former Daiichi-Kangyo Bank, and the up-to-date information systems of the former Fuji Bank. So, the system could handle transactions from all over the world all day and probably almost all night. So, you can imagine the kind of systems projects they had to go through. It must have been very tough, so when I heard of their 2002 system outage, I was sympathetic to their systems programmers. They probably had many sleepless nights. But, that wasn’t the last disaster. They and probably anyone probably never thought of receiving such large number of deposits concurrently without stop. Well, it happened.
A few executives probably have to resign from their posts to take responsibility. For the government, this event has been a good opportunity to go in and investigate the bank’s internal operations closely so that they can make sure the bank cannot fail to pay all the taxes even if their systems collapse.