In a recent presentation at The Ned (former Midlands Bank) just down the road from Bloomberg’s new European headquarters in the city of London, we learned why Bloomberg chose Apstra over other, better known intent-based networking technology companies.
The
occasion was doubly interesting because of Apstra’s reputation as a company to
watch, and because the Bloomberg installation marked a departure from the big
data center deployments that have made Apstra’s name to date. The company’s founder, David Cheriton, has been called “The
Professor Who Wrote Google Its First Check” – not quite true but, with a net
worth of $6 billion (Forbes 18/2/18), David Cheriton could very well be the
world’s richest academic. His contributions to education include $25 million
for graduate studies and research in the School of Computer Science at the
University of Waterloo, $7.5 million to the University of British Columbia, and
a $12 million endowment to Stanford University.
He is also an expert in distributed systems and networking, with exceptional insight into identifying big market opportunities. Yes, he was among Google’s first investors, and he has funded at least 20 companies including early investment in VMware, and Arista where he was co-founder and chief scientist.
Not just an angel, Cheriton also has pioneering experience in building the architectures needed to drive digital transformation and lower TCO. Not one to mince his words, in his keynote speech at NetEvents Global Press and Analyst Summit last year, David Cheriton explained how developers are tempted to be “slightly dependent” on the hundreds of services offered by AWS to save time but “later users find they can’t get off of AWS without re-writing the application.” For big companies that can add up to a million or more dollar costs per month, so enterprise customers want the ability to own and operate their own compute and networking infrastructure, but with the same agility, scalability, and cost as they get from the public cloud.
And this was the real reason why we were here – to hear how Apstra had successfully simplified Bloomberg’s network management and automated the company’s IT operations. Bloomberg is leveraging Apstra Intent-Based Analytics and Root Cause Identification, where a single source of truth is leveraged to identify and validate the root cause of network problems before it hampers performance. With real-time validation and remediation, the company is one step closer to their ultimate goal of becoming a self-operating network.
A striking demonstration of how vendor agnostic Apstra Intent-Based Data Center Automation can be applied not just to benefit massive telco, cloud provider, and Enterprise data centers but any highly complex and critical network.
The context
Bloomberg’s
spectacular European headquarters were designed to support a capacity of
thousands of people and thousands of devices, distributed work pods and meeting
pods with a central reservation system, as well as support for the Bloomberg
broadcast TV network and corporate network. The building has also won a number
of awards including being rated the world’s most sustainable office building.
We learned
that Bloomberg Television is Bloomberg Media’s global business and financial
television network, available in more than 437 million households worldwide.
The network streams live on Bloomberg’s digital and mobile properties, along
with Apple TV, Amazon Fire TV, Android TV, Samsung TV and Roku.
If Bloomberg is the central nervous system of global finance, then their broadcast television media production network must be the central nervous system of Bloomberg Media. A world-class network infrastructure was needed to support the availability, reliability, and performance needed for eight hours of live television and four hours of live radio every day.
The new media production network connects the central studio, three other broadcast positions, two recording studios, a photography studio and three control rooms and all of Bloomberg TV’s production equipment. The all-IP data network includes 90+ network nodes, 400+ 40GbE network fabric links, 4,000+ network ports, and 2,000+ live network endpoints connecting at a combination of 10GbE, 40GbE and 100GbE – not to mention a 14 petabyte video storage system.
Tony Lott, Bloomberg Global Head of Capital Projects for the company’s IT infrastructure team, along with Bloomberg Project Leader Inderpal Khattran, saw the new facility as an opportunity to create from scratch a radical state-of-the-art network. And their vision was to deliver it on time and within budget to support the today’s operations and well into the future.
They opted
for spine-leaf topology to provide consistency of data volume, scale and deterministic
application performance. Bloomberg
Television’s reputation depends on providing a seamless broadcast experience
for a worldwide audience of millions, so they wanted automation to continuously
check, monitor, and validate the health of the network. The need for greater
insight into the network fabric created a long checklist of stringent
compliance requirements, including a dashboard that showed the health of the
network and compliance to the stakeholders across the Operations, Security and
Compliance departments. They needed a single platform able to collect all the
telemetry from a variety of switch models and vendors in an agnostic manner, to
normalize the data and store it in such a way that the engineering teams could
query it in real-time.
Bloomberg Television runs its own broadcast TV playout across one of the spine-leaf fabrics within the campus building. So the IT department wanted the equivalent capability for the multicast traffic on that network – a notoriously difficult task, as the multicast network’s state is constantly moving and changing.
Finding a solution
Bloomberg
Television’s IT team looked at several solutions and combinations. Adding
multiple tools would become costly and cumbersome, but even so, they could not
match all their network objectives. They turned to an intent-based, vendor
agnostic solution: Intent-Based Analytics is not just about gathering
telemetry, it’s about processing the telemetry using a sophisticated big data
pipeline and comparing it against a single source of truth (i.e.
“intent”) to validate that the networking meets intent across general
health, performance, traffic and compliance.
They found
that no other solution could match the Apstra Intent-Based Data Center
Automation solution. It not only could ensure the general health and
performance of the network, the Apstra Operating System (AOS) could be
programmed to verify that the network met compliance at all times – for
example, to certify that all new software going into their environment met the proper
criteria for compliance.
Mansour Karam, Apstra’s Founder and CEO, explained the initial task: “When we came in to the Bloomberg building, the network was already there. So, what we had to do was discover the state of the network, the topology of the network, the capabilities of the network, with our solution. [AOS has] this network discovery capability that is critical for brown-field environments so that we can come into an already-existing environment that we leveraged and enhanced in order to deliver on Bloomberg’s requirements and deploy our solution in their environment”.
One of Bloomberg Television’s requirements was that the AOS agents run off-box. Karam explained: “When we launched the solution initially, we had on-box agents. These agents ran on the devices. That’s how we had it packaged, but it wasn’t a requirement of the architecture.” He added: “We have a distributed architecture with our product where it’s a very modular architecture. Every service, every piece of logic, is implemented as a separate process, including those agents in which they’re experts at the devices that the solution controls. In fact, this concept of agents is critical to the ability for our solution to be multi-vendor, because these agents are the experts at those devices.” So, for Bloomberg, they were readily able to run the agents off-box as required.
The solution was secured and certified for Bloomberg Television’s data center and soon proved its ability to continuously validate operator’s intent, while providing an early warning indicator of potential grey failures. An immediate benefit was that a number of existing network issues were immediately pinpointed – issues that no-one had yet detected. Such visibility into grey failures and Root Cause Identification is critical for such a complex and changeable environment – zero downtime is an absolute must for a broadcast TV network.
Reaping the reward
We learned
just how vital it was to optimize operational efficiency and provide immediate
answers to detailed questions about the network’s health and performance. Tony
Lott’s team can now take preventative action long before issues become
noticeable to the production staff.
The team has gained a single point of reference for all their network compliance needs. According to Tony Lott: “Apstra’s system gives us a single view into the entire network – regardless of our devices, switches, operating systems or vendors. It has enabled us to gain the day-2 operations efficiency needed to manage and sustain the complex network infrastructure in our cutting-edge facility.” It has already helped to reduce the company’s OPEX by almost half.
I do not think many of us had considered just how vital the back-end network infrastructure, security, or compliance would be to power a seamless broadcast TV experience for millions of viewers around the world. As Karam said: “This is a very modern, state-of-the-art building… Because of its stringent requirements, the decision was to have a network in there that is datacentre-class, very similar to the type of networks you have in datacentres. So, while it was a deviation from our market segment focus, the technology was very aligned with what our technology was capable of doing”.
There must be many other industries where network architects and engineers are facing similar challenges daily – and technologies originally inspired by the needs of hyperscale data centers now offer remarkably elegant solutions. That is partly why this deployment is so significant. As Karam pointed out: “What we’re seeing across the globe is that new campuses, new buildings, are really transforming… There is this concept of smart buildings – with IoT, with digital transformation, with the requirement for sustainability – putting a much bigger load on the network as the foundation to deliver on those requirements as part of those state-of-the-art modern campuses.”
Lott, Karam
and the Bloomberg IT team would agree: this surely is the way to go.