From Despair To Hope: 168 Hours Of Panic In The League

"We were saved from this" natural disaster ", but unexpectedly, a" man-made disaster "made us in a very difficult position. In February 27th, Sun Taoyong, the founder of micro League, said that the micro union is facing the biggest crisis in the company's history.

In February 25th, the micro Union issued a voluntary announcement at the HKEx to announce the news of the destruction of the production environment and data of the company's SaaS business after 19 p.m. in February 23rd. In February 28th, the micro union resumed all the production environment of SaaS business, but did not retrieve all the backup data. It only restored the backup data of the micro station products, and the data were recovered only at 20 in March 1st.

If this part of the data can not be recovered, the impact on the micro alliance will be hard to imagine. The micro Union, founded in 2013, landed at the Hong Kong Stock Exchange in January 2019. According to its mid 2019 financial report, SaaS business accounted for 33.4% of total revenue and 48.5% of gross profit.

In fact, the micro union encountered not only the data loss caused by staff misoperation, but a man-made malice. Earlier, a multi industry insider told reporters that under such circumstances, it is almost impossible to completely restore such massive data, but the micro alliance is lucky. From data loss to data recovery, micro Union went through 168 hours of panic.

Data disappear

At 6 p.m. on February 23rd, the micro union server issued a failure report. After investigation, it found that all servers were in a state of service failure. Then, the technicians restarted one of the servers and found that all the data in the system was gone.

"This time the situation has been very tense, because the server is a large area of failure, there may be two reasons, one is invasion, the two is deliberately destroyed." Xu Yongzhou, director of Tencent Yun Yun Wei and customer service department, said.

Tencent cloud then arranges two business lines for further investigation. One line is the server team. They continue to check the server, and finally determine that all the files on the micro union server have been lost. The other line is the security team. They cooperate with the technicians of the micro union to locate an authorized account into the database and delete the operation.

People who do technical work sometimes jokingly refer to "RM -rf / *", which is a file deletion command under the Linux system. It will delete all files including system and data, and it is not recoverable.

And the micro alliance is encountering this black swan event. It is reported that in addition to the database being deleted, the backup server can not log in, which means that the backup data is also deleted. The reason is that a core operation and maintenance staff of the company deleted the relevant business data.

Data recovery is actually a very professional field. In normal terms, it is led by enterprises and cloud manufacturers as a help. But at that time, there were no more ways for micro League to rely on Tencent cloud, and Tencent faced with this situation.

"Hundreds of T data, even backup is deleted, and want to recover in a relatively short time, which has not happened in the past history. We consulted several data recovery companies, and they all said that the situation can restore about 20% of the data is very good. Xu Yongzhou said frankly.

Emergency rescue

Generally speaking, data recovery work can be divided into three stages: the first step is to control the damaged surface, which is to ensure that those servers that have the opportunity to retrieve data do not have any problems; the second step is to retrieve data through professional software; the third step is data validation and on-line debugging.

Among them, the key is the first two steps, if the first two steps can be successfully completed, then the data will be recovered smoothly. This micro Union data recovery work is basically in accordance with this process.

According to Xu Yongzhou, according to the first step, they first need to mirror the source data to protect the source data. But through the analysis of the micro alliance situation, they feel that no matter the traditional network copy or hardware copy, they can not satisfy the timeliness and security.

As a result, Tencent cloud team made a bold decision: to cross the mirror copy steps, install another system disk to the original server, and scan the server directly through the new system disk loading OS and data recovery software.

"Many of our colleagues were unable to get to the scene while we were in operation. A group of people confirmed the steps in front of us through video links." Xu Yongzhou recalled that the whole process was very intense because it was an irreversible process and could not be reversed by any wrong step.

Fortunately, in the end, there is no danger. The reading of the source data is completed successfully, and the next challenge is data extraction. In February 26th, Tencent cloud team got the first batch of data, which was also normal after import and verification, which made them very excited and felt that victory was at hand. But they soon found that these data were up to February 17th. Even though the data were complete, there was still one week's data loss.

The mood of the whole team fell to the bottom again. But in fact, a series of operations have proved that the data still exist, but all that data is expected to be recovered.

As a result, Tencent cloud team has reformulated the "salvage" program. This time, they decided to scan every disk (block) of the disk according to the recovery plan of the data company. Meanwhile, the Tencent cloud team also developed a B plan, that is, to enter the salvage data from the OS layer of the server.

"B plan is a backup plan. It requires massive trial and data validation, which is time-consuming. Once the B plan is needed, the data of the micro League will not be completed as expected, and the specific time will be difficult to ascertain. Xu Yongzhou said.

Fortunately, the first block (block) of the first server was scanned successfully, and the data obtained was verified to be complete. This shows that the A plan is feasible. However, the size of data obtained by this way is far less than the size of the core data of the micro Union. This is like a huge puzzle that has been disrupted.

"The larger the data, the greater the difficulty of splicing. But fortunately, the backup mechanism of the micro League is relatively complete, and the data type is relatively unified. Finally, through a series of technical means, the splicing work has finally been completed, and the data of the micro League have been recovered successfully. Xu Yongzhou said.

In March 2nd, the micro Union announced that as of March 1, 2020, at 8 p.m., with the help of Tencent cloud, the data of the company's backup had been recovered, and the data will be resumed on March 2nd from 2 a.m. to 8 a.m., and the data will be resumed online at 9 a.m. from March 2nd 10 to March 3rd.

On the morning of March 3rd, the micro Union announced that as of March 3, 2020 at 6 a.m., the company's SaaS business data had been restored online.

Ring alarm bell

Loss of data and loss of micro alliance. After the official disclosure of data loss in February 25th, the price of micro Union fell sharply for three consecutive days, from HK $6.18 to HK $4.8, or over 20%. For businesses, micro Union has prepared 150 million yuan of compensation funds, and provides two kinds of compensation methods: cash compensation and traffic compensation.

An information security expert told reporters that human factors are now the main fuse that triggers enterprise data security problems, some of which are a misoperation of employees, resulting in server crashes or the deletion of core databases.

In response to this problem, Xu Yongzhou told reporters that enterprise data security can not rely solely on which strategy alone, but to have a complete set of systems, and every link can not be negligent. Compared with traditional data management, data security will be more secure on cloud.

For example, through the control of cloud account management authority, the access permission of the account resources can be controlled by granularity, such as allowing some sub accounts to have a read permission of a COS bucket, while another account has the write permission of the COS storage object. This can greatly reduce the risk of data corruption and loss caused by improper operation or unnecessary operation.

?