More than just VAR: How Sony’s Sports Businesses are driving the data-driven digital transformation of sport

Wimbledon is widely regarded as the most prestigious of tennis’s four Grand Slam tournaments, a lofty status assured by the All England Lawn Tennis Club’s (AELTC) commitment to tradition. The pristinely manicured grounds, all-white kit and strawberries and cream are as much part of the appeal as the world class sport on display.

Technology has played a key role in maintaining this prestige, whether it’s through broadcast or digital innovation, as well as improving the quality of sport on the famous grass courts. Since 2007, Wimbledon has let players challenge a line umpire’s decision using Hawk-Eye – a computer vision-based camera system that generates a virtual replay of the previous point with pinpoint accuracy.

Although in most cases the umpire’s decision is proved to be correct, the system makes the sport fairer and prevents lengthy arguments. The technology has enhanced television broadcasts and has given fans in the stands a new tradition to embrace. Fans slow clap as they wait for the replay to be shown on the big screen, a ritual that is now firmly part of the in-person experience. 

Hawk-Eye’s ball-tracking system is now the de facto industry standard in tennis, including at the US Open, where the Hawk-Eye Live system is used to power automated line calling on every single court at Flushing Meadows in New York. 

Across all sports, the company has partnerships with 23 of the world’s top 25 rights holders. It powers cricket’s decision review system (DRS), delivers tracking services to Major League Baseball (MLB), and provides video capabilities to the National Football League (NFL). 

In soccer, Hawk-Eye has helped engineer two of the biggest changes in soccer since the sport was codified in the late 19th century with Goal Line Technology (GLT) and Video Assistant Referee (VAR). It now plans to be at the forefront of the next great innovation in in-pitch technology – semi automated offside technology (SAOT).

Hawk-Eye believes the skeletal tracking technology that enables SAOT will help it transition from a technical supplier into a truly collaborative partner for the sports industry, helping partners maximise their investments in officiating technology to create new data-driven revenue streams.

Sony’s technology powered ESPN and Disney’s Toy Story Funday Football simulcast (Image credit: Disney)

SkeleTRACK

Hawk-Eye’s proprietary SkeleTRACK system uses cameras located around the side of the pitch to track all 29 skeletal points on all 22 players on the pitch at a rate of 50 times a second. This data is combined with ball-tracking technology and means SkeleTRACK knows the precise location of every single joint and limb on the field at any single moment in time.

Should it detect that any part of a player interfering with play is in an offside position, an alert will be sent to the video match officials who will then validate the decision using a video replay. If the decision is upheld, a notification is sent to the referee’s watch in a similar fashion to GLT, and a 3D virtual recreation of the incident is displayed on the big screen and on the television broadcast for viewers at home.

The primary function of SAOT is to produce more accurate offside decisions at speed, reflecting Hawk-Eye’s central mission of improving sport. The duration and deliberation process of offside decisions has been one of the main criticisms of VAR, with relatively little information given to fans in the stadium. Meanwhile, the current method of using lines to determine which part of a player’s body might be offside is deemed unsatisfactory and often confusing.

“Instead of a referee having to consult VAR or go to a referee screen [to review the decision], which could take more than a minute, this new system is significantly quicker,” explains Rufus Hack, chief executive of Sony’s Sports Businesses. “Instead of drawing the lines to see if a player is offside, which many people don’t like, we can create a virtual recreation of the situation with sub-second latency because we are tracking all of the data.

“You effectively build a virtual wall, like the one in the goal with GLT, and you can see the limbs poking through. This is going to be much easier for fans to accept visually – much like in tennis when you can see the ball is either in or out.

“I think this a real gamechanger for the sport. We have industry leading video replay technology and now we increasingly have industry leading tracking technology.”

The power of skeletal tracking

The underlying technology has a whole host of potential use cases for leagues, federations and clubs to maximise their investment and use the data to boost fan engagement and accelerate digital transformation.

Because SkeleTRACK is tracking the entire field of play in real time, it is effectively creating a data-driven digital twin of the pitch that can be repurposed into a whole range of content – from data visualisations for digital platforms to an entirely new form of second screen experience. 

If the entire field and the entire match is reproduced in 3D, fans can replay any incident from any angle, unimpeded by sightlines or the decisions of the broadcast match director. It will make the free roam camera from the instant replay in the FIFA video game series a reality. Such capabilities can add value to existing rights, be used in first-party applications, or even create an entirely new rights class that could be sold to bookmakers. 

“We’ve actually been working with Serie A on this,” says Hack. “What we can do is virtually recreate the game with sub-second latency so fans can watch the game without being restrained by the camera angle. They can focus on a different player such as the goalkeeper, or review incidents in real time. They can have their own data overlays. It just becomes a much more flexible and immersive experience.

“Rights holders can effectively create a new asset class which doesn’t cannibalise media rights. This could be very, very attractive for the betting community to increase dwell time and get more engagement on their sites than if they just had a data feed and are unable to acquire live streaming rights.”

SOAT was first used in the Uefa Champions League earlier this season and was showcased to a global audience at the 2022 Fifa World Cup in Qatar. But Hawk-Eye sees SAOT as just one use case of skeletal tracking, believing the technology provides an unprecedented level of data-driven visibility that can overhaul everything from fan engagement through to performance.

“I think this a real gamechanger for the sport. We have industry leading video replay technology and now we increasingly have industry leading tracking technology.”

Rufus Hack, CEO, Sony Sports Businesses

Going beyond as Sony’s Sports Businesses

That mission is exemplified by the fact that Hawk-Eye is now just one part, albeit a very important element, of Sony’s Sports group. The group also includes digital sports media specialist Pulselive, which counts the likes of the Premier League, the International Cricket Council (ICC) and World Rugby among its clients, and Beyond Sports, whose visualisation engine and artificial intelligence (AI) capabilities can help rights holders create entirely new digital experiences for fans in the metaverse. 

“There is a real opportunity for Sony to be the strategic partner for the world’s top-tier rights holders as they make this transition from B2B to B2C and own the relationship with the customer,” explains Hack. “Sony believes there is a convergence across multiple entertainment genres and there’s a real commitment at the highest level of the company to invest in sport.”

Pulselive is an essential component in this end-to-end vision, helping rights holders use assets from previously siloed systems and divisions to create something greater than the sum of its parts. The company already works with some of the biggest names in sport, creating engaging and innovative digital platforms that maximise this treasure trove of sporting and fan data. Most sports fans have used a Pulselive-developed product, such as an official fantasy game, without even knowing it.

The data and video gathered by Hawk-Eye’s officiating technology can be transformed into entirely new experiences through Beyond Sport’s data visualisation capabilities and distributed directly to fans using Pulselive’s digital expertise. The group is also uniquely positioned to  leverage innovations from Sony’s other entertainment businesses, particularly in gaming, music and entertainment.

“Hawk-Eye has grown up trying to make sport better,” continues Hack. “We’ve tried to improve the quality, accuracy and fairness of sport and we have good, trusting relationships with the industry that give us an opportunity to become a real strategic partner.”

“Pulselive is regarded as having the best technological backbone, the best CMS, and the best new fan data platform within the industry. But we don’t just want to help rights holders build incredibly robust, aesthetically pleasing digital platforms. We want to drive engagement and create communities that accelerate direct-to-consumer (DTC) strategies.

“We can help sport disrupt and diversify by adapting to the macro changes we’ve seen in sports content by leveraging technological investments to create new, immersive content that drives fan engagement.”

Beyond Sports technology is capable of creating entirely new forms of digital content using skeletal tracking data (Image credit: Sony)

Unlocking new possibilities

Hack believes this combination will drive innovations in data-driven content for broadcast and digital platforms, as well as in fantasy sports, interactive experiences, the metaverse and more. 

The potential for gamification is an obvious benefit of being part of the wider Sony ecosystem given the capabilities of PlayStation. Hack suggests rights holders could let fans apply custom avatars into the second screen experience so they can ‘play’ the same game they are watching on television or make different substitutions to the coach and compare the outcomes.

Such an experience could prove critical in attracting Gen Z audiences who consume sport predominantly through digital platforms and have grown up with gaming as a core component of their sporting experience.

Beyond Sports’ technologies can take real time data collected from the field of play by SkeleTRACK to create new types of social content, power interactive simulcasts of live events, and facilitate metaverse use cases that will be the future of how fans consume sport. A great example of this in practice was Nickelodeon’s ‘blocky’ livestream of the 2021 NFL wildcard game, which won a Sports Emmy award. These experiences can exist either as a linear broadcast or as an immersive experience on a VR headset, becoming an entirely new way to watch sport. 

Most recently, Beyond Sports powered Disney and ESPN’s ‘Toy Story Funday Football ‘ a fully animated simulcast of the NFL contest between the Jacksonville Jaguars and Atlanta Falcons featuring beloved characters and imagery from Pixar’s beloved movie series.

THE CLAW IS SPOTTING THE BALL

🏈 Toy Story Funday Football on Disney+ pic.twitter.com/DsaXThRnth

— NFL (@NFL) October 1, 2023

Other parts of the Sony group are also investing heavily in the metaverse, creating technology that can be applied to its sporting efforts. Sony is already working with City Football Group (CFG), the owners of Premier League side Manchester City and several other soccer teams around the world, on a project that recreates the Etihad Stadium in an interactive, digital environment for fans across the globe to explore.

“You can visit the club museum, go into the changing rooms, and step out onto the pitch,” Hack explains. “At some point we’d like to be able to let fans watch the match in a virtual matchday experience using all that data.

“Sony Music have been hosting concerts in the metaverse for five years so the technology is already there, we’re just repurposing it.”

All of this shows how the expertise offered by Sony’s sports businesses, combined with the resources of the wider group, can be used to create digital experiences that would be impossible in isolation. 

There’s even a performance angle. Hawk-Eye’s technology has tracked thousands of MLB games, providing teams with data that helps manage pitcher workloads and take preventative action that mitigates the threat of injury that can curtail their season.

“The teams use this data in real time to analyse things like whether the pitcher is over-rotating their rotator cuff, risking injury, to help decide whether to pull him from the game,” explains Hack. “We’ve also been talking to some Premier League clubs about how they can use skeletal data to infer tiredness levels through stats like stride length. We’ve actually brought on a team of data scientists to start understanding and analysing these metrics and thinking about what the potential performance benefits are.”

Beyond this, there is hope the technology can help reduce and identify instances of concussion, helping to address one of the most pressing issues in high-impact collision sports.

Hawk-Eye has championed many technologies that have fundamentally altered how sport is played, but now the group wants to change how sport is experienced. It wants to use the trust and relationships it has cultivated, the digital expertise of Pulselive, and Beyond Sports’ vision of the future to further the digital transformation of sport. 

“We can offer the strategy, content, and technology to help rights holders make their core sport better, safer and fairer,” concludes Hack. “But we can also leverage the investments they’ve made to do that to create new, immersive content that can help them drive their fan engagement agenda.

“I think with the Sony brand behind us, we’re incredibly well positioned to do that.”

Sony’s Sports Businesses have already changed how sport is played, officiated and consumed, and there is more to come later this year.