Top Of The Peak: Tencent'S "Spring Festival Defense War"

In the Spring Festival of 2021, many people did not get together with their families on New Year's Eve for the first time under the call of the whole country to celebrate the new year's Eve. For Zou Fangming, new year's Eve has long been a common practice in the company. For the past nine years, he spent his new year's Eve with his colleagues.

In the past, Zou Fangming and his colleagues used to call Haidilao's takeout to the company. They ended the year with a hot pot. But this year, because they moved to a new building, they couldn't open fire, so they had to go to a restaurant opposite the company to eat.

The so-called new year's Eve dinner, the whole process is very dull, no one drinks, the meal time is only 5:30 to 6:30 in the afternoon, about an hour. After the meal, everyone quickly returned to work. Because if there is no accident, they will usher in a wave of user peak at 7 p.m., and the next time, their task is to monitor wave after wave of user peak, and deal with the possible sudden traffic.

With the ringing of the new year's bell, all the monitoring software in front of Zou Fangming was normal. They could finally breathe a sigh of relief because they had survived the "spring protection" for another year.

Zou Fangming is now the director of self research business center of Tencent infrastructure department. The main responsibility of his department is to provide research and development support for Tencent's internal self research business. In short, the Department has access to more than 90% of Tencent's business, and the underlying technical support of Tencent products users usually use, such as wechat and QQ, is related to it.

Zou Fangming told the 21st century economic report that Tencent's products are mainly social products, and there must be hot events, critical moments and key protection moments in social products. Among them, the Spring Festival is the most challenging and important protection time.

It is reported that Tencent started to set up the "spring insurance" project from the Spring Festival in 2013. Xiao Zhili, vice president of Tencent cloud and general manager of Tencent infrastructure department, said in an interview that the peak of spring festival actually existed in the era of PC Internet, but at that time, it was not so convenient for users to surf the Internet, so the peak was not on New Year's Eve, but on the fourth or fifth day of the first day. After the Spring Festival, we should upload some photos to QQ space for classification But on the whole, the security pressure was not great at that time.

However, since 2012, especially after the emergence of wechat, the wave of mobile Internet has swept in, and the communication and interaction between users has become more real-time, which also gives birth to the real spring festival peak. The reason why this is a real peak is that compared with the past pc internet era, its peak has increased hundreds of times. Of course, this huge contrast also left many painful memories for the technical support staff.

Spring insurance

For Zou Fangming, 2013 was his first experience of "Chunbao", which was also his most "painful" year.

At that time, the wechat released in January 2011 was developing rapidly. In March 2012, the number of wechat users exceeded 100 million, and by the beginning of 2013, the number of wechat users had exceeded 300 million. Zou Fangming is the R & D director of wechat chat pictures and chat video system. As mentioned above, before 2013, Tencent did not have a clear concept of "spring protection", and Zou Fangming had never encountered such a fast-growing product. Therefore, in his mind, he did not think that wechat would suddenly appear "big congestion".

On the new year's eve of 2013, under the background of the whole mobile social networking in full swing, people's interaction on the mobile terminal becomes more and more frequent. In wechat, users hold up various group chats and start sending new year's greetings videos in groups. The charm of social products lies in its fission, which spreads from one generation to the next, and from ten to one hundred. However, for technicians, this fission brings terrible high concurrency.

Zou Fangming recalled that on New Year's Eve in 2013, the number of new year's Eve videos suddenly increased by six or seven times, which has exceeded their reserve. At that time, in the face of the sudden huge amount of video, Zou Fangming was at a loss because he had no preparation plan and could not temporarily access more server resources. In order to protect the whole system from collapse, they could only take the method of degradation.

The downgrade here refers to limiting the size of video users can send. At first, the limit is that videos above 10 megabytes are not allowed to be sent. Later, it becomes 6 megabytes, 4 megabytes and 2 megabytes. The whole process of degradation lasts from 7:00 p.m. to 10:00 p.m., which lasts for three hours. Zou Fangming said that at the peak time of 8:00 p.m., they almost limited 70% - 80% of users to send videos, which was very terrible. During this period, they also received a large number of complaints from users, but they were really at a loss. They could only survive the traffic peak through such a crude way.

After the painful experience of the Spring Festival in 2013, Tencent began to make preparations for "spring protection". Zou Fangming introduced that the preparatory cycle of the "spring protection" project basically starts from October every year and lasts for about two to three months. During the preparation process, the product and technical team will first determine the product playing methods during the Spring Festival, and then measure how much bandwidth and storage each product needs according to these playing methods.

Of course, at this time, it is not enough for the products to meet the demand for resources. Zou Fangming's team will also focus on balancing the demand and resources. The purpose is to ensure the user experience of each product through technology upgrading and flexible strategies in the case of limited resources.

However, in the Spring Festival of 2015, under the preparation of "spring protection", accidents still happened. Because in October 2014, wechat launched the small video function, which can be shared with friends circle and chat interface. Therefore, on the eve of "spring protection", that is, in October and November, the number of users of wechat small video was not large, and the capacity was given dozens of times according to its magnitude.

But on New Year's eve of 2015, wechat small video suddenly produced a huge peak that exceeded expectations. About an hour before the Spring Festival Gala began, the number of small video releases increased sharply, and the system began to malfunction. Zou Fangming and his colleagues began to expand the capacity urgently. However, since the wechat small video system was used to reuse Tencent video, it was more inclined to PGC than UGC, so the processing process was more complicated.

Zou Fangming recalled that after the emergency expansion of an index module, the index was ok, but as the volume continued to rise, there were problems with the front-end access module, and after handling the access module, another video processing module failed.

"So that night, the problem continued for about three hours. After we expanded all the places that could be expanded, the user peak was over." Zou Fangming said that although the whole system has not been affected, but for users, this new year's Eve video experience is not good. So after the Spring Festival in 2015, Tencent Infrastructure Department adjusted the whole video system architecture and optimized the scene of small video.

Bear the pressure of red envelope

In the Spring Festival of 2015, in addition to the emergency brought about by wechat small video, Tencent's "Chunbao" actually ushered in another big test -- the interaction of wechat red envelope Spring Festival Gala.

Xiao Zhili told reporters that in November 2014, they learned from the wechat payment team that the Spring Festival Gala and the Spring Festival Gala would cooperate to carry out a red envelope activity, and the number of users reached would be very large. At that time, it was only two months away from the Spring Festival, which was undoubtedly a major test for Xiao Zhili's team.

Xiao Zhili said that there were three main problems: first, the original red envelope system was definitely not available, because the original system was used for tens of thousands of people inside the company. Now, if it is to be expanded to one billion users, the whole system must be redone.

Secondly, the Spring Festival red envelope activity is broadcast by mouth in the Spring Festival Gala to let users participate in the red envelope snatching. The challenge is that it is difficult to estimate how many visits the system will carry. "I don't know how much pressure we have to bear. This is actually a big taboo to do architecture, but we did not have any reference at that time." Xiao Zhili said.

Finally, the technical support work is only allowed to succeed, not to fail. "If it fails, it will be a disgrace in front of the people of the whole country, so the pressure on the technical level will be greater than ever."

In less than two months, Xiao Zhili, with the red envelope system team of Tencent infrastructure department and the wechat payment team, stayed all night in Shenzhen Tencent building. The version was submitted at 12:00 p.m. every day, the test started at 1:00 a.m., and the results came out at about 3:00 a.m., and everyone continued to study how to optimize and transform the next day.

Until about February 10, 2015, the whole red envelope system was basically determined. But until three days before New Year's Eve, the whole team was still changing the code, because in the process of repeated testing, Xiao Zhili would ask the team to change it if it was found that there might be risks. "We must try our best to eliminate all the risks."

Finally, Tencent's 2015 Spring Festival Gala red envelope activity was successfully completed, and wechat payment also completed the catch-up with Alipay in the field of mobile payment. According to official wechat data, on New Year's Eve 2015, the total number of wechat red packets received and received reached 1.01 billion times. From 8:00 p.m. to 0.48 a.m. of the next day, the total amount of wechat shaking interaction reached 11 billion times.

Xiao pan, director of the technology operation center of Tencent infrastructure department, told the 21st century economic report that for the red envelope system, if it encounters a huge traffic pressure beyond expectations, they will also take some measures to degrade, such as making some restrictions on the amount of money, so as to quickly reduce the concurrency.

In addition, according to Xiao pan, the whole red packet system is like a funnel, and each layer is filtering. Therefore, the "snatching" of red packets is the highest user concurrency, followed by disassembly. When the real transaction is finally completed, the user concurrency is much less.

Although in recent years, the development speed of wechat red packets is also very fast, but there will be no leap growth from 100000 to 1 billion as in 2015. At present, the number of users of wechat red packet is close to that of wechat, and the growth of concurrency during the Spring Festival is relatively stable. For Tencent's red packet system, the main task now is to continuously optimize the system and reduce costs and resources.

"Chunbao" becomes normal

Zou Fangming told reporters that if we review Tencent's "spring insurance" in the past nine years, it can be roughly divided into two stages. The first stage is from 2013 to 2016, which can be called the stage of insufficient bearing capacity.

In this period of time, there are the reasons for the rapid development of business, but also the reasons for Tencent's own technical architecture is not perfect. In short, the system capacity is not enough to support the growth of business, leading to various failures. Therefore, in this period, the main work is to improve the carrying capacity. In addition to increasing hardware equipment, in terms of technical means, it is also necessary to continuously improve the performance.

The second stage is after 2016. In this stage, with the precipitation and Optimization in the past few years, Tencent's bearing capacity has basically no problem. At this time, we need to improve. With the increasing requirements of users for product experience, Tencent also needs to continuously optimize the carrying capacity on the basis of ensuring the normal operation of the system. For example, in the past, the technical requirement was that the denial of service should not exceed 30 minutes, but now, the denial of service will not happen, and the technical requirement is that the service delay should not exceed 5 minutes.

This year's "spring protection", Xiao Zhili's overall experience, feeling very smooth, all peaks are within the forecast range. Therefore, for the Infrastructure Department of Tencent led by Xiao Zhili, the pressure of the support work at key moments such as "Chunbao" has tended to be stable under the existing business conditions. What we should pay attention to are some sudden critical moments.

For example, at the beginning of 2020, due to the new epidemic situation, Tencent meeting, which was launched at the end of 2019, ushered in a critical moment. According to the original goal of Tencent Conference team, it is expected that dau will reach 50000 by the end of 2020. By the middle of February, Tencent's online users at the same time have exceeded 4 million, reaching a 1000 fold increase.

This substantial growth has also brought huge challenges to Tencent's infrastructure department. Xiao Zhili said that in fact, from the technical level, Tencent has a lot of business with millions of online users, and it is not a difficult problem to support this kind of business. But the reason why the situation was urgent at that time was that under the influence of the epidemic, they could not find enough physical resources. For example, the BGP broadband of the operators could not meet the demand, the IP address of the external network could not meet the demand, some equipment suppliers could not return to work, and all kinds of equipment could not be supplied.

At this time, these hardware problems can only be solved through software. Xiao Zhili said that if the IP address is not enough, we should try to use the technology of port reuse. If the new machines can not be supplied, we should mobilize servers from all over the country and re adapt them through virtualization or software architecture to strive to support Tencent's Conference business first.

It is in this extremely difficult situation that Tencent has completed the expansion of 1 million nuclear power. In fact, from the difficulty point of view, the emergency expansion of Tencent conference is more difficult than dealing with the red envelope activities of the Spring Festival Gala in 2015, because it is a challenge to cope with without any preparation. The reason why Tencent can withstand the pressure is because it has experienced many important moments like "spring protection" in the past.

Zou Fangming said that now from the frequency of failures and the stability of maintenance, technical support work has become easier, because everyone has experienced many battles, and many employees have been very familiar with the performance and requirements of each business since the "spring insurance" work in 2013. So now the big situation is mainly because I didn't expect it, but in fact, this kind of unexpected situation has become less and less.

?