The story of the “evolution” from the digital transformation process at Viettel’s Global Operations Center is a vivid demonstration of that fact.
At Viettel, the Global Operations Center, Viettel Network Corporation (VTNet) is one of the leading units in the network. The special thing is that this unit is not only responsible for operating and ensuring the quality of Viettel’s network and service quality in Vietnam, but also has the entire Viettel telecommunications and information technology infrastructure in 10 foreign markets. outside.
In order to maintain the stability of Viettel’s network in each market, a system of hundreds of people from the Global Operations Center takes turns on duty 24/7 to handle any situation from small to large. arising from Viettel network system from Asia, Africa to America.
Where the lights never go out
The Global Operations and Operations Center’s Network Operations Room is a place where the lights are almost never turned off. This is an important “link” in Viettel’s centralized network management and operation system – Global Network Operation Center (GNOC).
At the Global Operations Center in Viettel, the lights never go out.
As the “operator” of the network of more than 110 million customers in 11 countries, the GNOC system monitors and influences inter-provincial and national network systems. As a monitoring satellite, GNOC allows technical experts in Vietnam to also grasp the status of each BTS station in Peru, Haiti, Tanzania… despite the great geographical distance.
Due to special requirements, in foreign markets, Viettel’s technical force is very thin. Meanwhile, the field of telecommunications is too broad, so one person cannot understand many areas and even less deeply. Helping markets with quality control, troubleshooting, operation and remote network impact for many years has been undertaken by VTNET.
Thanks to the ability to operate the network remotely, Viettel’s large telecommunications systems in foreign markets do not need too many engineers, helping to deploy quickly, save costs and increase efficiency.
During the period of 2020 – 2021, when the Covid-19 pandemic broke out, Viettel was one of the businesses that made the strongest contributions in terms of both resources and technology to support the Government, the health sector and localities nationwide. response to the pandemic.
In that context, the workload of the Center skyrocketed due to a series of large-scale anti-epidemic support technology solutions that were run on Viettel’s infrastructure such as the tracking system (Bluezone), medical declaration (the newspaper). health declaration, Unified health declaration system), health information provision/management (University health, Health records), National QR code system, vaccination support system (Book electronic health), a video conferencing system for disease prevention and control….
Many engineers have to eat and rest for a whole month at the Center to ensure the distance between epidemic prevention. In that context, the Center responded quickly and effectively to epidemic prevention and control activities as well as ensuring important national and international political events such as the US-North Korea Summit.
Up to now, the Center has ensured the infrastructure for more than 2,300 video conferencing points, 571 isolation points, absolutely safe control of 6 online meetings run by the Prime Minister to fight the epidemic to the commune level…
“You have to learn to code and change your mind!”
According to Mr. Le Quang Hieu, Director of the Center, at VTNet from the day it was established to the time Covid-19 occurred, about 40% of the Center’s personnel were in the “on-call” group. This operating model is arranged in the direction of specialization, each employee can cover and monitor all 11 markets at the same time. Although the detailed description system has helped to automate the processes, when something goes wrong, the alert staff will create tickets to operate the system, forward the required orders to the solving function.
Mr. Le Quang Hieu, Director of Viettel Global Operations Center: “To overcome the disadvantages of repetitive work, we asked you to learn programming and change your mindset. “.
For this job, the personnel in charge only need to receive processes and tools from other specialized departments and comply with 100%, ensuring no problems. The pure job is to monitor the quality of the network and system during on-call duty, if any problems arise, report them to superiors in accordance with the procedures and regulations.
The passive and repetitive nature of work makes you less likely to develop yourself. Moreover, the division of 24/7 shifts also affects health and spirit. “In order to overcome these disadvantages, we asked you to learn programming and change your mindset,” recalled Mr. Hieu.
The Center actively selects young and potential actors to learn programming. When there is a request to adjust the tool, the center’s staff will directly evaluate and make timely corrections, instead of waiting for feedback from all levels or the programming team. In addition, this team also actively and comprehensively researches systems, finds root causes of problems and risks, and offers solutions to improve the sustainability, security and safety of the network.
Optimizing human resources thanks to the “flattening” strategy
Along with the changes at the individual level, the Center’s leaders also set a strategic goal of optimizing personnel with the project of Shifting remote operations. Before that, the Center had nearly 600 people, also known as Viettel’s Super Center by leaders.
The central feature at the moment is divided into many layers, divided into 1 core network, 3 areas, 6 processes and a tool ecosystem. The Operations Center is in charge of the core network, each North, Central and South region will have its own NOC (Network Operation Center) forces to be on duty and in charge of the provincial and regional access layer.
In the process of operation, the multi-layer model of management reveals the disadvantage of dispersing management and technical forces. Problems and problems that arise will have to go through many complicated processes and procedures to be thoroughly resolved. Along with the increasing demand for network and information technology system development, the need to optimize personnel and processes is becoming urgent.
It is from this fact that the Center decides to “flatten”, applying a single troubleshooting process to make it more streamlined. “With this approach, for example, in Hanoi, it is also possible to know the problem in Can Tho, helping to reduce the time to handle and fix the problem for customers to ensure quality and the shortest and fastest time possible. can,” shared Quang Hieu.
Deploying to remove the tiered operating model, switch to “flattening” the Center has consolidated the entire area and all personnel to the Center. Instead of 3-4 operating layers, there is now only one layer left. The entire network in Vietnam will only have 1 NOC force in charge of the entire core network and 3 regions instead of 4 forces as before.
“The number of NOC personnel has decreased by 20% so far and will continue to decrease further in the future. Many former NOC personnel have now been transferred to other creative and more proactive jobs. “, said Mr. Hieu.
Change yourself to reach the world
Thanks to the efforts to optimize and improve the quality of human resources, in 2021, the Global Operations Center will leave many imprints when pioneering scientific and technological research, positioning the role and image of Viettel’s brand in the industry. national and international.
Within Viettel, the Center sets itself to challenge requirements, further promote the process of digitization, automation, and comprehensively improve the quality of operation and exploitation: replacing manual action basically with new technologies researched and developed by Viettel engineers such as ChatBot, Cloud Monitoring, Auto Escalate, One Click VMSA… and reduced 40% of network incidents, no subjective serious problems.
On a national level, the Center has made a great contribution in technology to the prevention and control of the Covid-19 epidemic through research and implementation of the latest solutions for infrastructure optimization and information security applied to systems. Bluezone, PC-Covid, Electronic Health Book, Medical Declaration… help the systems operate efficiently and safely.
In the international arena, the fact that the Center’s operational engineers have been honored by global organizations has helped Viettel contribute to the world’s open source platform, ranking 44th out of 157 vendor contributions to OpenStack and ranking No. 1 Vietnam as well as Southeast Asia. Viettel also plays a key role in coordinating the Open Source symposium in the Internet Day 2020 event.
The successful lesson of the Center, summarized by Director Le Quang Hieu, is “Dare to think differently, renew yourself, pioneer transformation”. Mining operations engineers actively participate in R&D, go to programming classes and directly join the development force to accelerate progress.
“The success of the Center is due to the consensus to change from the plans, tools, solutions and even the process from the Board of Directors to the technical brothers below,” explained Director Le Quang Hieu.
Success thanks to fast Cloud technology
One of the outstanding technologies that contributes to the success of the Center is the cloud computing infrastructure owned by Viettel, capable of supporting quick, convenient and economical service deployment.
During the pandemic, Viettel proactively provided solutions along with infrastructure requirements for emerging technology needs. Thanks to the effective support of Viettel’s Cloud service, the time to meet the needs of infrastructure resources for the above important projects, systems and products has been significantly shortened.
“With systems of a similar size, it used to take several weeks for Viettel to solve it, but now it can confidently meet these needs in just a day, even with a few days. can be completed within 24 hours and still guarantee the whole process”, said Cloud Operations and Deployment Engineer Trong Vinh, Global Operations Center.
In terms of technology, Viettel has also gradually and completely mastered Cloud technology, and at the same time built complete and automatic deployment and monitoring toolkits. These technologies all meet the strictest standards in the world. In particular, it is impossible not to mention the automation toolkits developed by Viettel to effectively serve the implementation and operation tasks.
According to VTNet leaders, in the field of cloud computing, in the past, Vietnam was only in the role of “beneficiary” of available technologies in the world. Now, the story is how to develop that core technology and whether Vietnam can master it or not. “Viettel’s operation and exploitation engineers have been deploying cloud infrastructure since 2018. Up to now, in addition to the story of only bringing world technology to deploy, we discovered the errors and directly participated in discussions with them. engineers on the world forum, changed from passively receiving and asking questions to actively doing,” said Mr. Le Quang Hieu.
This, according to the leader of VTNet, helps the level of Viettel engineers to rise very high, very quickly and is recognized by the world. Currently with 17,000 servers, ranked in the top 19% of the world’s leading cloud enterprises, VTNet’s cloud infrastructure is among the top in Southeast Asia. Especially, most of the system was built and developed by Viettel’s talented team of young 9x generation engineers.
Leave a Reply