The videos from the past are a thing of the past, but now, thanks to the artificial intelligence that not only serves to humiliate us playing Go or create amazing (and not so amazing) deepfakes, also serves to restore videos and images of the past, and make them look as if they had just been recorded yesterday.
Videos from the past: this is how life was more than a century ago
These kinds of old videos we are referring to, started appearing shared in social networks recently. We are talking about the legendary 1896 Lumière Brothers video that showed a train arriving at a station. Next you can see the original video remastered, the video rescaled and softened and finally the video also with color applied automatically with the development of a powerful application called DeOldify and which we will talk about later.
A Russian programmer called Denis Shiryaev was the one who started all this, applying two types of process on a video already remastered in the year 2017. The first process consists of 4K scaling through Topaz Labs’ Gigapixel AI platform. The second process consists of the generation of interpolated frames and is known as DAIN. This second process was created by Google engineers and the aim of this technological application is to add more frames to the videos, in order to make the movement of the images more smoothly.
And so Shiryaev continued to “create” more of these spectacular videos, such as this amazing journey through the streets of New York in 1911 from a remastered video in 2018 that was originally captured by the Swedish company Svenska Biografteatern and that even preserves the sounds of the time.
As Shiryaev himself explains, besides the process of ‘upscaling’ to 4K with an ESRGAN and a smoothing to 60 FPS thanks to the mentioned DAIN, he used After Effects and some plugins to improve the definition and details, and finally applied the neural network proposed by DeOlfidy to colour these images, in short, to make the videos from the past look as if they had been recorded today.
The result is simply prodigious, and this type of technique has been applied to more and more videos from that early era of cinematography.
In his Youtube channel, Shiryaev offers some videos from the past taken in San Francisco in 1906, Amsterdam in 1922, England in 1901 or Moscow in 1896, and in the same way, he has recomposed such interesting videos as the one of the Apollo 16 moon rover ride.
The interpolation that gives rise to this rescaling or overscaling has been better known for some time and artificial intelligence systems have helped to speed up and improve the process, but things have become even more interesting when it comes to colouring these images and getting from black and white to quite convincing colours.
DeOldify creates color where almost there was none
Jason Antic (https://twitter.com/citnaj), a software enthusiast, began researching so-called Generative Adversarial Networks (GAN) in early 2019. As explained in NVIDIA’s Podcast AI, after completing an online course at Fast.ai to learn about how neural networks work, he began a unique project: trying to apply this branch of artificial intelligence to a very specific and concrete task: converting black and white images to full color images.
His work in this field ended up giving a spectacular result: the DeOldify project – with the code in GitHub for anyone to use – not only colors images, but restores them. However, it should be noted that it is certainly not the only project in this sense.
The project not only works for single images, but it is also possible to apply it to videos through the so-called NoGAN, a new type of GAN system that reduces the training of the neural network and achieves spectacular results that correct errors of the pure approach applied by DeOldify.
The result of this work can be applied to all kinds of videos from the past. During the Facebook F8 conference, Antic demonstrated the result applied to a small excerpt from the 1960 film ‘Psycho’, but its application extends to all kinds of images and videos, and has been used to – among other things – transform those old videos from the early 20th century and make them enjoyable today with surprisingly convincing colour results.
In a long interview with Humans of Machine Learning, Antic revealed many of the keys to his work in this field. He had never tried to restore the color of a photo in Photoshop, for example, and while studying his course in artificial intelligence he realized that the flaw in automatically coloring images is that there always had to be a human reviewing them to see if everything was okay.
That made the neural network very conservative in its predictions, he explained, but with the GANs you could do a realistic coloring with much less effort and that despite being more “risky” gave much better results. Even so, he stressed that “there is no single color that is valid for many things (like clothes, for example),” so part of the process is “art, and that’s probably why it’s difficult to get a neural network to do it right.”
This area is also the focus of the work of Robert Cross (@RobCross247), an expert in design and digital architecture who has long maintained a project in which he transforms old black and white photos of Ireland into colour, with fantastic results thanks to DeOldify.
This process, which until not long ago was done basically with image post-processing programs such as Photoshop, has advanced considerably thanks to artificial intelligence. Another extensive post in FloydHub analyzes the process in this case. While the results may be “magical”, the reality is that there is a lot of science behind them.
Turning black and white images into color has become for some an interesting hobby, and in fact subreddits like Colorization or ColorizedHistory have become a unique museum with the results that users achieve using this kind of process. Many of them are impressive and show that this technique is already quite advanced.
A process that is partly an art
Beyond the debate on whether this process betrays the original content or not, the truth is that the images offered by these methods are spectacular and make that visit to the past much more shocking for all audiences because, thanks to the treatment of these videos and images of the past, that ancient time is made more present than ever.
Although the techniques for achieving video and image enhancement are absolutely new, their application is almost a small art in itself. Many of them are processed with developments like DeOldify, but some are even touched up by hand. The result, in any case, is simply magic, as shown in the following twitter thread.
1. New York City 1911 part. 1 pic.twitter.com/lq75O8r8mu
— Joaquim Campa (@JoaquimCampa) April 17, 2020
As Denis Shiryaev, who presents his work in Neural.love, explains with that 1906 San Francisco video (days before the terrible earthquake that struck the city) restored, there are a few components in action in that transformation.
- Noise reduction: to achieve images without those traditional mistakes of old movies.
- Sharpening: to improve the level of detail and contours of all elements displayed on the screen.
- Frame interpolation (FPS boosting): to go from 24 or 30 to the 60 frames per second that achieve that smoothness of the videos, something that is particularly well explained in this project by Wenbo Bao.
- Color: achieved with DeOldify and which as he points out is not necessarily historically accurate but provides that impressive transformation from black and white to color.
- Rescaling to 4K: to make videos, captured at a much lower resolution and even remastered at higher resolutions, end up being shown natively in 4K with a spectacular result.
- Audio: many of these videos also had audio recorded, but also needed to be processed to increase the quality and definition.
Anyone can do it but a beastly GPU helps a lot
The algorithms, as Shiryaev explained, are still slow and need powerful hardware to process more or less quickly. Modern graphics cards are increasingly helping in this area, and in fact NVIDIA has been pushing this type of application for a long time in both its previous GTX 1000 family and in the new RTX 2000.
Antic himself warned about this on the DeOldify project page:
To be able to do these tasks with some ease, it’s highly recommended to have a super-powered graphics card
Antic says it would love to have more than the 11 GB of graphics memory it has in its GeForce 1080 Ti for the training phase, although for the coloring you can use more modest cards with about 4 GB of graphics memory.
GitHub explains very well how the whole process works, but Antic has also prepared a guide with documentation that allows you to start taking the first steps in a task that is certainly surprising. So, if you want to cheer up there’s a good way to do it and now is a good time to try, since, free time is what’s left in these days of quarantine.