An Interview with BBC R&D’s “AI in Production” team - winners of IBC’s Best Paper Award 2018
Craig Wright, Jack Allnut, Rosie Campbell, Michael Evans, Ronan Forman, James Gibson, Stephen Jolly, Lianne Kerlin, Zuzanna Lechelt, Graeme, Phillipson and Matthew Shotton
Why is the BBC interested in AI for production?
As we describe in the paper, for several years BBC R&D has experimented with ways to create video coverage of events that are out of scope for conventional outside broadcasts. The Edinburgh Fringe is a good example, with tens of thousands of performances at hundreds of little venues across the city. Bringing shows like this to new audiences, using small, unobtrusive camera rigs, is our first use case for AI in Production. And that’s brilliant, because the potential impact of AI in our industry is so profound that it might have been hard to know where to start. The opportunity here in terms of audience and creative value was so clear that our process of understanding the ways that AI and ML (machine learning) might change our industry could start with the creation of a working prototype, which we’ve called “Ed”. It’s given us a chance to learn by doing and to build our skills in machine learning, in AI, and in working with real creatives to understand the sort of rules an automated system needs.
Please could you tell us about the team and their range of skills?
The team’s been a multidisciplinary one since the project started, and that’s been successful and very rewarding. Our software and algorithmic engineering is informed by human-centred research science. The initial rules coded into the Ed system, describing things like how to compose a well-framed shot, came from observing and talking with programme makers, and understanding the psychology and the teaching of cinematography principles. To iterate and improve on our work we’re doing empirical, subjective studies with real people. Our technical skills cover a number of relevant areas, including production automation, data science, video fundamentals, human-computer interaction and neuroscience.
How do you find recruiting engineers / graduates with AI/ML skills?
The key to successful automation is an in-depth understanding of the problems to be solved, so rather than bring in AI/ML experts and teach them about our industry, we see this project as an opportunity to build up these valuable data science skills within the BBC’s existing workforce. This benefits the BBC, our project and our colleagues. For the future, we’re forming partnerships with university groups that specialise in this area - getting access to their expertise, and offering our industry’s challenges as opportunities for the next generation of AI engineers.
What is your view on the long-term impact of AI on our industry?
Increasing automation has been a theme in our industry for decades now - AI is just the latest example. We think that the application of AI to media production will have many benefits, such as bringing many more live events to audiences, and helping creative professionals with mundane and repetitive tasks. These will be good outcomes if, as an industry, we keep our eyes on the really important things - like creative and editorial standards, craft skills development, representation of diversity, and creative vision. We can’t do what we’re doing in AI in Production, without the important work that is going on in parallel in BBC R&D and across the rest of the organisation. We’ve had colleagues working with the UK Parliament Committee on AI, and leading critical work in areas like ethical data-centric technologies for society.
Do you use ML based feature extraction tools in Ed and are they robust enough?
ML techniques aren’t inherently less robust than any other algorithms. We’re looking at a wide range of ML and other AI methods to improve Ed. Our engineering approach is to test and benchmark the robustness and performance of any combination of algorithms so that we can understand their fitness for our purposes. It’s clear, though, that it will probably be a long time before an AI-based system can approach the ability of humans to handle unusual and unexpected scenarios.
Beyond the panel show, what other live genres would you like to develop for Ed?
We’re very aware that it’s relatively easy to train machine learning algorithms to do specific tasks, and much harder to synthesise more general intelligence. When we’ve trained the Ed system to be good at comedy panel shows, it’s going to be fascinating to see how well it does with things that are superficially similar—like musical performances or monologues—but which differ in terms of what the algorithms need to look for and what the audience expects to be shown.
What’s next for your team?
There’s so much to learn in this area; we’d like to build a community of people in industry and academia who can see the potential that AI and machine learning offer to the field of media production. We’d like to expand the capabilities of Ed to look at genres beyond panel shows, and we’d like to do more “bottom up” research, looking at the challenges being faced by real production teams (typically the most boring and repetitive tasks) and asking how AI could help them work more efficiently, and allow creative professionals to spend more time focussing on the creative aspects of their work. Our IBC paper this year concentrates on full automation, but we hope that our next one will help tackle the (potentially much harder problem) of skilled craftspeople working in collaboration with AI.
In what timeframe might we expect to see a live broadcast programme shot and edited with AI assistance?
Our current algorithms are significantly slower than real time. Their speed will definitely improve but it’s almost certain that data-driven systems like Ed will always produce better quality output if they can analyse the whole performance - looking forward and backwards in time. Our team is most interested in coverage that’s ‘nearly live’; made available within an hour or two of the live performance, and hopefully people will be able to see content like that within the next year.
But we aren’t the only team in BBC R&D looking at AI. Earlier this year colleagues worked with BBC Four on an experiment using ML to schedule an evening's output on the channel. Algorithms searched out content from the BBC archive that would be relevant to BBC Four's audience and highlight it to the scheduling team.