AI-Generated 'Mask' Revolutionizes Art Restoration, Completing Work in Hours Instead of Years

Have a damaged painting? Restore it in just hours with an AI-generated "mask"

Art restoration takes steady hands and a discerning eye. For centuries, conservators have restored paintings by identifying areas needing repair, then mixing an exact shade to fill in one area at a time. Often, a painting can have thousands of tiny regions requiring individual attention. Restoring a single painting can take anywhere from a few weeks to over a decade. In recent years, digital restoration tools have opened a route to creating virtual representations of original, restored works. These tools apply techniques of computer vision, image recognition, and color matching, to generate a "digitally restored" version of a painting relatively quickly. Still, there has been no way to translate digital restorations directly onto an original work, until now. In a paper appearing today in the journal Nature, Alex Kachkine, a mechanical engineering graduate student at MIT, presents a new method he's developed to physically apply a digital restoration directly onto an original painting. The restoration is printed on a very thin polymer film, in the form of a mask that can be aligned and adhered to an original painting. It can also be easily removed. Kachkine says that a digital file of the mask can be stored and referred to by future conservators, to see exactly what changes were made to restore the original painting. "Because there's a digital record of what mask was used, in 100 years, the next time someone is working with this, they'll have an extremely clear understanding of what was done to the painting," Kachkine says. "And that's never really been possible in conservation before." As a demonstration, he applied the method to a highly damaged 15th century oil painting. The method automatically identified 5,612 separate regions in need of repair, and filled in these regions using 57,314 different colors. The entire process, from start to finish, took 3.5 hours, which he estimates is about 66 times faster than traditional restoration methods. Kachkine acknowledges that, as with any restoration project, there are ethical issues to consider, in terms of whether a restored version is an appropriate representation of an artist's original style and intent. Any application of his new method, he says, should be done in consultation with conservators with knowledge of a painting's history and origins. "There is a lot of damaged art in storage that might never be seen," Kachkine says. "Hopefully with this new method, there's a chance we'll see more art, which I would be delighted by." Digital connections The new restoration process started as a side project. In 2021, as Kachkine made his way to MIT to start his PhD program in mechanical engineering, he drove up the East Coast and made a point to visit as many art galleries as he could along the way. "I've been into art for a very long time now, since I was a kid," says Kachkine, who restores paintings as a hobby, using traditional hand-painting techniques. As he toured galleries, he came to realize that the art on the walls is only a fraction of the works that galleries hold. Much of the art that galleries acquire is stored away because the works are aged or damaged, and take time to properly restore. "Restoring a painting is fun, and it's great to sit down and infill things and have a nice evening," Kachkine says. "But that's a very slow process." As he has learned, digital tools can significantly speed up the restoration process. Researchers have developed artificial intelligence algorithms that quickly comb through huge amounts of data. The algorithms learn connections within this visual data, which they apply to generate a digitally restored version of a particular painting, in a way that closely resembles the style of an artist or time period. However, such digital restorations are usually displayed virtually or printed as stand-alone works and cannot be directly applied to retouch original art. "All this made me think: If we could just restore a painting digitally, and effect the results physically, that would resolve a lot of pain points and drawbacks of a conventional manual process," Kachkine says. "Align and restore" For the new study, Kachkine developed a method to physically apply a digital restoration onto an original painting, using a 15th-century painting that he acquired when he first came to MIT. His new method involves first using traditional techniques to clean a painting and remove any past restoration efforts. "This painting is almost 600 years old and has gone through conservation many times," he says. "In this case there was a fair amount of overpainting, all of which has to be cleaned off to see what's actually there to begin with." He scanned the cleaned painting, including the many regions where paint had faded or cracked. He then used existing artificial intelligence algorithms to analyze the scan and create a virtual version of what the painting likely looked like in its original state. Then, Kachkine developed software that creates a map of regions on the original painting that require infilling, along with the exact colors needed to match the digitally restored version. This map is then translated into a physical, two-layer mask that is printed onto thin polymer-based films. The first layer is printed in color, while the second layer is printed in the exact same pattern, but in white. "In order to fully reproduce color, you need both white and color ink to get the full spectrum," Kachkine explains. "If those two layers are misaligned, that's very easy to see. So I also developed a few computational tools, based on what we know of human color perception, to determine how small of a region we can practically align and restore." Kachkine used high-fidelity commercial inkjets to print the mask's two layers, which he carefully aligned and overlaid by hand onto the original painting and adhered with a thin spray of conventional varnish. The printed films are made from materials that can be easily dissolved with conservation-grade solutions, in case conservators need to reveal the original, damaged work. The digital file of the mask can also be saved as a detailed record of what was restored. For the painting that Kachkine used, the method was able to fill in thousands of losses in just a few hours. "A few years ago, I was restoring this baroque Italian painting with probably the same order magnitude of losses, and it took me nine months of part-time work," he recalls. "The more losses there are, the better this method is." He estimates that the new method can be orders of magnitude faster than traditional, hand-painted approaches. If the method is adopted widely, he emphasizes that conservators should be involved at every step in the process, to ensure that the final work is in keeping with an artist's style and intent. "It will take a lot of deliberation about the ethical challenges involved at every stage in this process to see how can this be applied in a way that's most consistent with conservation principles," he says. "We're setting up a framework for developing further methods. As others work on this, we'll end up with methods that are more precise." This work was supported, in part, by the John O. and Katherine A. Lutz Memorial Fund. The research was carried out, in part, through the use of equipment and facilities at MIT.Nano, with additional support from the MIT Microsystems Technology Laboratories, the MIT Department of Mechanical Engineering, and the MIT Libraries.

Years of painting restoration work done in just hours by new technique

At left is a damaged painting, with the middle panel showing a map of the different kinds of damage present - green lines show full splits in the underlying panel support, thin red lines depict major paint craquelure, blue areas correspond to large paint losses, while pink regions show smaller defects like scratches - at right is the restored painting with the applied laminate mask Although AI-based restoration methods can indeed bring new life to damaged paintings, the end result is typically a digital copy of the original painting. By contrast, a new MIT technique applies reversible repairs to the physical painting itself, in the form of a removable mask. The process was developed by mechanical engineering graduate student Alex Kachkine, who restores paintings via traditional hand-painting techniques as a hobby. He realized that many galleries have a number of paintings which never get displayed, because they require restoration that would take too long - and thus be too expensive - to perform by hand. Utilizing his method, however, restoration times could be reduced from years, months or weeks down to a matter of hours. When developing his system, Kachkine started out with a highly damaged 15th century oil painting that he owned. Using traditional cleaning methods, he removed all the extra paint that had been applied over the years in past restoration efforts. He then performed a high-resolution scan of the cleaned painting. Next, Alex utilized existing AI algorithms to analyze that scan, creating a digital model of what the painting likely first looked like in its pristine, unblemished form. He subsequently used his own software to create a map of the painting which identified all the places where the original paint had faded, cracked, or was otherwise damaged. That map also indicated the exact colors that needed to be applied to those locations, in order to restore the painting to its original appearance. Utilizing a commercial high-fidelity inkjet printer, that digital map was converted into a physical two-layer mask printed onto an ultra-thin clear polymer film. One layer was printed in the required colors - in their required locations - while the other layer was white. "In order to fully reproduce color, you need both white and color ink to get the full spectrum," he explains. In a final step, the mask was carefully aligned with and applied to the surface of the painting, then adhered in place with a thin spray coating of varnish. Importantly, both the mask and the varnish can be dissolved/removed with existing conservation chemicals if desired, without damaging the original paint. Additionally, the digital map will serve as a permanent record of the changes that were made to the original, for reference by future conservators. For Alex's project, a total of 57,314 different colors were used to repair 5,612 separate regions of the painting, in a process that took only about three and a half hours. He estimates that performing the same task solely by hand would take 66 times longer. "There is a lot of damaged art in storage that might never be seen," says Kachkine. "Hopefully with this new method, there's a chance we'll see more art, which I would be delighted by."

Researchers create AI-based tool that restores age-damaged artworks in hours

By slashing time and cost of restoration, technique could be used on paintings not valuable enough for traditional approach The centuries can leave their mark on oil paintings as wear and tear and natural ageing produce cracks, discoloration and patches where pieces of pigment have flaked off. Repairing the damage can take conservators years, so the effort is reserved for the most valuable works, but a fresh approach promises to transform the process by restoring aged artworks in hours. The technique draws on artificial intelligence and other computer tools to create a digital reconstruction of the damaged painting. This is then printed on to a transparent polymer sheet that is carefully laid over the work. To demonstrate the technique, Alex Kachkine, a graduate researcher at Massachusetts Institute of Technology, restored a damaged oil-on-panel work attributed to the Master of the Prado Adoration, a Dutch painter whose name has been lost, as a late 15th-century painting after Martin Schongauer. The painting is extremely detailed but visibly split into four panels, covered in fine cracks and dotted with thousands of tiny patches where paint has fallen off. "A lot of the damage is to small, intricate features," said Kachkine, who estimated it would have taken about 200 hours to restore the painting with traditional conservation techniques. "It has undergone centuries of degradation." Kachkine started with a scan of the painting to determine the size, shape and position of the damaged areas. This identified 5,612 separate sections that needed repair. A digital mask was then constructed in Adobe Photoshop. To restore missing specks of paint, spots were added and colour-matched to surrounding pigments. Damage to patterned areas was corrected by copying similar patterns from elsewhere in the painting. The missing face of an infant was copied from another work by the same artist. Once finished, the mask was printed on to a polymer sheet, varnished to prevent the ink from running and overlaid on the painting. In all, 57,314 colours were used to infill damaged areas. The corrections are designed to improve the painting, even if they are not perfectly aligned. On seeing the result, Kachkine was delighted. "It followed years of effort to try to get the method working," he said. "There was a fair bit of relief that finally this method was able to reconstruct and stitch together the surviving parts of the painting." The approach, described in Nature, can only be used on varnished paintings that are smooth enough for the sheet to lie flat on. The mask can be peeled off or removed using conservators' solvents, leaving no traces on the original artwork. Kachkine hopes the method will allow galleries to restore and display scores of damaged paintings that are not deemed valuable enough to warrant traditional restoration. But he acknowledges there are ethical issues to consider, such as whether having a film covering a painting is acceptable, whether it hampers the viewing experience, and whether particular corrections, such as copied features, are appropriate. In an accompanying article, Prof Hartmut Kutzke at the University of Oslo's Museum of Cultural History, said the approach provided a way to restore damaged paintings faster and more cheaply than was possible with conventional techniques. "The method is likely to be most applicable to paintings of relatively low value that would otherwise be housed behind closed doors, and might not be suitable for famous, valuable artworks," he said. "However, it could widen public access to art, bringing damaged paintings out of storage and in front of a new audience."

Have a damaged painting? Restore it in just hours with an AI-generated 'mask'

Art restoration takes steady hands and a discerning eye. For centuries, conservators have restored paintings by identifying areas needing repair, then mixing an exact shade to fill in one area at a time. Often, a painting can have thousands of tiny regions requiring individual attention. Restoring a single painting can take anywhere from a few weeks to over a decade. In recent years, digital restoration tools have opened a route to creating virtual representations of original, restored works. These tools apply techniques of computer vision, image recognition, and color matching, to generate a "digitally restored" version of a painting relatively quickly. Still, there has been no way to translate digital restorations directly onto an original work, until now. In a paper appearing in Nature, Alex Kachkine, a mechanical engineering graduate student at MIT, presents a new method he's developed to physically apply a digital restoration directly onto an original painting. The restoration is printed on a very thin polymer film, in the form of a mask that can be aligned and adhered to an original painting. It can also be easily removed. Kachkine says that a digital file of the mask can be stored and referred to by future conservators, to see exactly what changes were made to restore the original painting. "Because there's a digital record of what mask was used, in 100 years, the next time someone is working with this, they'll have an extremely clear understanding of what was done to the painting," Kachkine says. "And that's never really been possible in conservation before." As a demonstration, he applied the method to a highly damaged 15th century oil painting. The method automatically identified 5,612 separate regions in need of repair, and filled in these regions using 57,314 different colors. The entire process, from start to finish, took 3.5 hours, which he estimates is about 66 times faster than traditional restoration methods. Kachkine acknowledges that, as with any restoration project, there are ethical issues to consider, in terms of whether a restored version is an appropriate representation of an artist's original style and intent. Any application of his new method, he says, should be done in consultation with conservators with knowledge of a painting's history and origins. "There is a lot of damaged art in storage that might never be seen," Kachkine says. "Hopefully with this new method, there's a chance we'll see more art, which I would be delighted by." Digital connections The new restoration process started as a side project. In 2021, as Kachkine made his way to MIT to start his Ph.D. program in mechanical engineering, he drove up the East Coast and made a point to visit as many art galleries as he could along the way. "I've been into art for a very long time now, since I was a kid," says Kachkine, who restores paintings as a hobby, using traditional hand-painting techniques. As he toured galleries, he came to realize that the art on the walls is only a fraction of the works that galleries hold. Much of the art that galleries acquire is stored away because the works are aged or damaged, and take time to properly restore. "Restoring a painting is fun, and it's great to sit down and infill things and have a nice evening," Kachkine says. "But that's a very slow process." As he has learned, digital tools can significantly speed up the restoration process. Researchers have developed artificial intelligence algorithms that quickly comb through huge amounts of data. The algorithms learn connections within this visual data, which they apply to generate a digitally restored version of a particular painting, in a way that closely resembles the style of an artist or time period. However, such digital restorations are usually displayed virtually or printed as stand-alone works and cannot be directly applied to retouch original art. "All this made me think: If we could just restore a painting digitally, and affect the results physically, that would resolve a lot of pain points and drawbacks of a conventional manual process," Kachkine says. 'Align and restore' For the new study, Kachkine developed a method to physically apply a digital restoration onto an original painting, using a 15th-century painting that he acquired when he first came to MIT. His new method involves first using traditional techniques to clean a painting and remove any past restoration efforts. "This painting is almost 600 years old and has gone through conservation many times," he says. "In this case there was a fair amount of overpainting, all of which has to be cleaned off to see what's actually there to begin with." He scanned the cleaned painting, including the many regions where paint had faded or cracked. He then used existing artificial intelligence algorithms to analyze the scan and create a virtual version of what the painting likely looked like in its original state. Then, Kachkine developed software that creates a map of regions on the original painting that require infilling, along with the exact colors needed to match the digitally restored version. This map is then translated into a physical, two-layer mask that is printed onto thin polymer-based films. The first layer is printed in color, while the second layer is printed in the exact same pattern, but in white. "In order to fully reproduce color, you need both white and color ink to get the full spectrum," Kachkine explains. "If those two layers are misaligned, that's very easy to see. So I also developed a few computational tools, based on what we know of human color perception, to determine how small of a region we can practically align and restore." Kachkine used high-fidelity commercial inkjets to print the mask's two layers, which he carefully aligned and overlaid by hand onto the original painting and adhered with a thin spray of conventional varnish. The printed films are made from materials that can be easily dissolved with conservation-grade solutions, in case conservators need to reveal the original, damaged work. The digital file of the mask can also be saved as a detailed record of what was restored. For the painting that Kachkine used, the method was able to fill in thousands of losses in just a few hours. "A few years ago, I was restoring this baroque Italian painting with probably the same order magnitude of losses, and it took me nine months of part-time work," he recalls. "The more losses there are, the better this method is." He estimates that the new method can be orders of magnitude faster than traditional, hand-painted approaches. If the method is adopted widely, he emphasizes that conservators should be involved at every step in the process, to ensure that the final work is in keeping with an artist's style and intent. "It will take a lot of deliberation about the ethical challenges involved at every stage in this process to see how this can be applied in a way that's most consistent with conservation principles," he says. "We're setting up a framework for developing further methods. As others work on this, we'll end up with methods that are more precise."

Put the paintbrush down - AI can restore artworks quicker and better

Artificial intelligence (AI) could spell the end of art restoration by humans after MIT showed that damaged paintings can be repaired in just a few hours. Typically, conservators spend months or years researching and matching paints, colours and techniques to ensure the finished product is as close to the original as possible. But Alex Kachkine, an engineering graduate at MIT, has shown it is possible to use AI to fill in the damaged areas digitally, then print the restored layers onto a thin film to attach on top of the painting. It means the painting appears restored even though the original is still intact beneath. The method was applied to a highly damaged 15th-century oil painting, and AI immediately identified 5,612 separate regions in need of repair, and filled in these regions using 57,314 different colours. The entire process, from start to finish, took just three and a half hours. "There is a lot of damaged art in storage that might never be seen," said Mr Kachkine. "Hopefully with this new method, there's a chance we'll see more art, which I would be delighted by." In recent years, digital restoration tools have allowed conservators to create virtual representations of restored works. AI algorithms can quickly sift through huge amounts of data about artists and time periods to generate a digitally restored version of a particular painting, in a way that closely resembles the correct style. However, digital restorations are usually displayed virtually or printed as stand-alone works and until now there has never been a way to translate the digital restorations on to the original work.