AI Video Scraping: Extracting Data from Screen Recordings with Unprecedented Efficiency
3 Sources
[1]
Cheap AI "video scraping" can now extract data from any screen recording
Researcher feeds screen recordings into Gemini to extract accurate information with ease. Recently, AI researcher Simon Willison wanted to add up his charges from using a cloud service, but the payment values and dates he needed were scattered among a dozen separate emails. Inputting them manually
[2]
AI researcher scrapes usable data from a 35-second screen recording for less than one cent via Google Gemini
This could potentially save thousands of hours in manual labor. AI researcher and data journalist Simon Willison used the Google AI Studio tool to convert a 35-second screen recording of 12 emails into a single spreadsheet. This experiment surprised Willison, who did not expect the AI to return
[3]
This Expert Says AI Efficiency Soars When it Uses Video to Crunch Numbers
Willison was working on one of those everyday accounting tasks that sounds simple, but inevitably ends up being time consuming. He wanted to tally all the different charges he'd incurred for using a cloud company's services. But, as news site ArsTechnica notes, Willison's data was embedded all over
Share
Copy Link
AI researcher Simon Willison demonstrates a novel "video scraping" technique using Google's Gemini AI to extract data from screen recordings, potentially revolutionizing data collection and analysis.
AI Researcher Pioneers "Video Scraping" Technique
Simon Willison, an AI researcher and data journalist, has introduced a groundbreaking method called "video scraping" that utilizes artificial intelligence to extract data from screen recordings. This innovative approach could potentially save countless hours of manual labor and revolutionize data collection processes
1
.The Experiment
Faced with the tedious task of compiling charges from multiple emails, Willison devised a creative solution. He recorded a 35-second video scrolling through twelve relevant emails and fed this recording into Google's AI Studio tool, which provides access to various versions of Google's Gemini 1.5 Pro and Gemini 1.5 Flash AI models
1
.AI-Powered Data Extraction
Willison prompted the Gemini AI to extract price data from the video and arrange it into a JSON (JavaScript Object Notation) format, including dates and dollar amounts. The AI successfully completed this task, allowing Willison to easily convert the data into a CSV (comma-separated values) table for spreadsheet use
2
.Surprising Efficiency and Cost-Effectiveness
The accuracy of the results and the low cost of running the video model astounded Willison. The entire video analysis process used just 11,018 tokens on the Gemini 1.5 Flash 002 model, which would typically cost less than one-tenth of a cent. In this case, the process was free due to Google AI Studio's current promotional offering
1
.Implications and Potential Applications
This "video scraping" technique has far-reaching implications for data collection and analysis. It could be particularly useful when dealing with large volumes of data scattered across numerous sources. The method requires no specialized knowledge, making it accessible to a wide range of users
2
.Related Stories
Privacy Considerations
While this technique offers significant advantages, it also raises privacy concerns. Similar concepts are used in tools like Microsoft's Recall for Copilot+ PCs and the third-party Rewind AI tool for macOS. These tools continuously record screen activity, potentially making user data vulnerable, even if processed locally
2
.Future of AI Assistants
Willison's experiment hints at the future capabilities of AI assistants, which may soon be able to see and interact with users' on-screen activities. This could lead to more intuitive and efficient AI-human interactions in various fields, from data analysis to everyday computing tasks
3
.References
Summarized by
Navi
[2]
Related Stories
Recent Highlights
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
