How FreeBSD Servers Equipped with AMD EPYC CPUs Helps Netflix To Achieve 400Gbit/s Per Server
My Geek Score: This week, at the EuroBSD conference, in an effort to ensure high-quality video streaming to an audience of hundreds of people (with more potentially tuning in as the meeting goes on), Netflix deployed FreeBSD operating system servers with AMD EPYC 705 and 706 CPU architectures, running the AMD EPYC I/O Controller (IOC) software stack. The work, which had previously been done internally by Netflix, has been shared on GitHub for FreeBSD Project community members to get their hands dirty with.
Netflix’s FreeBSD Servers
A couple of years ago Netflix started looking for FreeBSD servers for streaming video. They noted that their upstream efforts were going no further, as most of their feedback focused on Debian GNU/Linux. Netflix had similar challenges, such as packaging the libraries in Python, and dealing with a lack of community support. They ended up using OpenBSD instead due to its former support for the VideoLAN Streaming Media Extensions (VLC) and the BSD-licenced OpenConnect framework. But they had another issue: FreeBSD did not support OpenConnect global network topology. They had to use a second protocol called AVIF to get their images served. Recently Netflix deployed its first OpenConnect based video service, which utilizes AMD EPYC CPUs on 12 of its FreeBSD VMs.
The Demonstration by Netflix
The demonstration was primarily used to showcase how well Netflix’s video streaming technology works. On one server, Drew Gallatin added HEVC support in order to better handle real-time tasks. “While that was a tremendous undertaking, the streaming videos were streamed very quickly,” said Drew Gallatin. “The other servers have totally different video encoders and decoders, however with the same basic functionality.” In terms of scale, Netflix can use 4 racks of servers for 8 million simultaneous users. Each server is capable of supporting 400Gbit/s. The Dx switch and the Mellanox port are used to connect the servers. Gallatin noted that while Netflix’s video streaming technology is capable of 500Gbit/s, the performance degradation would be too high for most applications.
What Is FreeBSD?
Introduced in 1991 as the BSD Unix, it is the kernel-based operating system derived from AT&T Unix. It is the ancestor of the open source BSD. Nowadays, FreeBSD powers Netflix, Google, Microsoft, and Amazon. It is the most popular Unix-based operating system for security-sensitive applications. It’s primarily focused on performance and reliability. This is why the OS is called a workhorse for enterprise computing. It is free, open source, and the code is openly available. There is no license fee. It is also highly customizable. Many add-ons are available, making it ideal for cloud computing. High Bandwidth requirements for a video streaming service are typical. One important feature of FreeBSD is the inclusion of the General Public Channel Interface (GPCI).
Why Are 400 Gbit/s per Server Significant?
As Gallatin pointed out in his tweet, Netflix needed to improve its system image streaming. So, the team came up with the 400 Gbit/s per server benchmark for purposes of “conveying how we’re benchmarking to Netflix. They are large and have high bandwidth requirements.” Netflix signed up with Gallatin in 2015 when he was still working for Akamai as the senior lead of Akamai’s Networking Research Group. He said in an interview with FOSS Force that one of the big challenges in determining the performance requirements for Netflix was how to integrate (HMM) to the client side. Gallatin said that while AMD is equipped to perform that, he was unsure whether OpenH264 would scale for the purpose of Netflix’s requirements.
How Many People Can Be Served by 400 Gbit/s?
During the demonstration, it was stated that it would take 400 of these servers to deliver 1 Gbit/s with 100GbE connections. The servers have a total of 32 cores and 96 GB RAM. If you look closely, it is easy to see that some of those servers are operating at around 40% load. With the exception of a few people streaming in the office, no one was using the streaming servers. Even if 100 people were using them, it would still take six to seven of these to deliver 1 Gbit/s. If you were running 400, that would be a dramatic improvement. What Are They Doing to Optimize Their Servers? Gallatin was able to compare the performance of these servers to FreeBSD’s capabilities.
Conclusion
It’s important to note Netflix worked with Gallatin and others to ensure its project was secure and reliable. A demo using a 1 Gbps connection and then 400 Gbit/s also featured a stable networking system. This project illustrates some of the unique advantages FreeBSD provides. Gallatin said the resulting 400 Gbit/s transfer rate is the best he’s seen for a closed network. Gallatin has worked on an open-source project to test his Meltdown mitigation solution. At least one other open-source project intends to solve the Spectre vulnerabilities, but Gallatin is a bit different from most of them. Gallatin believes a combination of mitigation and prevention will bring a fundamental change in the computing industry.
