OpenAI's O1 Preview: A Powerful New AI Model with Promising Capabilities and Potential Concerns

OpenAI's o1: The Good, the Bad, and the Ugly of AI's Latest Brainchild - Decrypt

Last week, OpenAI unveiled its latest AI model, o1, after a wave of speculation involving different post-GPT4 models with cryptic names including "Strawberry," "Orion," arguably "Q*," and the obvious "GPT-5." This new offering promises to push the boundaries of artificial intelligence with enhanced reasoning capabilities and scientific problem-solving prowess. Developers, cybersecurity experts, and AI enthusiasts are abuzz with speculation about o1's potential impact. In general, enthusiasts have praised it as a significant step forward in AI evolution, while others -- more familiarized with the model's insights -- have advised us to tame our expectations. OpenAI's Joanne Jang made it clear from the beginning that o1 "isn't a miracle model." At its core, o1 is the product of a concerted effort to tackle problems that have traditionally stumped AI systems, especially with inaccurate prompts that lack specificity. From complex coding challenges to intricate mathematical computations, this new model aims to outperform its predecessors -- and in some cases, even human experts -- particularly at tasks that require complex reasoning. But it's not perfect. OpenAI o1 grapples with performance issues, versatility challenges, and potential ethical quandaries that may make you think twice before choosing it as your new default model in the vast pool of LLMs to try. We have been playing a little bit with o1-preview and o1 mini -- maybe too little, since it only allows for 30 or 50 interactions per week, depending on model -- and we have come up with a list of the things we like, hate, and are concerned about regarding this model. o1's problem-solving capabilities stand out as its crowning achievement. OpenAI argues that o1 often surpasses human PhD-level performance in solving specific problems, mainly in fields like biology and physics, where precision is paramount. This makes it an invaluable tool for researchers grappling with complex scientific questions or large amounts of complex data. Zero-shot Chain of Thought One of o1's most intriguing features is its "Chain of Thought" processing method. This approach allows the AI to break down complicated tasks into smaller, more manageable steps, analyzing the potential consequences of each step before determining the best outcome. It's akin to watching a chess grandmaster dissect a game, move by move, or going through a reasoning session before making a decision. TL;DR: o1's logical and reasoning capabilities are outstanding. OpenAI o1 is particularly good when it comes to programming. From education to real-time code debugging and scientific research, o1 adapts to a wide range of professional applications. OpenAI paid special attention to o1's coding capabilities, making the model more powerful than its predecessors, and more versatile at understanding what users want before translating tasks into code. Other models are good at coding, but applying Chain of Thought to a coding session makes the whole process more productive, and the model is capable of executing more complex tasks. Jailbreak protection OpenAI hasn't overlooked the critical issue of ethics in AI development. The AI giant equipped o1 with built-in filtering systems designed to prevent harmful outputs. This may be seen as an artificial and unnecessary constraint by some, but for big businesses having to deal with the consequences and responsibilities of what an AI does in their name, they may be keen on safer model that doesn't recommend that people die, doesn't produce illegal content, and is less prone to being tricked into proposing or accepting deals that may result in financial losses. OpenAI claims that the model is designed to resist "jailbreaking," or attempts to bypass its ethical constraints -- a feature that's likely to resonate with security-conscious users. But we already know their beta efforts were not good enough, though perhaps the official release candidate will do better. The model exhibits sluggish performance, especially when compared to its speedier cousin, GPT-4o. In fact, the "mini" versions were specifically designed to be fast. This lack of responsiveness makes o1 less than ideal for tasks that demand rapid-fire interactions or high-pressure environments. Of course, the more powerful a model is, the more computing time it will take. However, part of this sluggish experience is due to the embedded Chain of Thought reasoning process that it has to go through before providing an answer. The model "thinks" for around 10 seconds before starting to write its answers. In our tests, some tasks have taken the model more than a minute of "thought time" before answering. Then, add another bit of time for the model to write a huge and long chain of thought process before giving you the simplest answer in the world. Not good, if patience isn't your strongest suit. It's not multimodal -- yet o1 also comes with a relatively stripped-down feature set. It's missing some functionality that developers have come to rely on in models like GPT-4. These absent features include memory functions, file upload capabilities, data analysis tools, and web browsing abilities. For tech professionals accustomed to a fully-loaded AI toolkit, working with o1 might feel like downgrading from a Swiss Army knife to something like a standard blade. Also, those using ChatGPT for more creative purposes are also locked out of the model's true potential. If you need DALL-E, OpenAI o1 won't work, and the powerful GPTs cannot benefit from using this powerful model either. It's text-only for now. OpenAI promised to integrate those functions in the future, but as it stands today, a text-only chatbot may not be the best option for most of its users. A text LLM only prompts what the image generator creates. Not integrating DALL-E with OpenAI o1 seems like a step back in terms of versatility. It sucks with creativity Yes, o1 is great at reasoning, coding, and doing complex logical tasks. But ask it to create a novel, improve a literary text, or proofread a creative story and it will fall short. OpenAI acknowledges this, and even in ChatGPT's main UI, it says GPT-4 is better for complex tasks and o1 is best for advanced reasoning. As such, its groundbreaking model is weaker than the previous generation when users require a more "generalist" approach. This makes o1 feel like less of an overwhelming leap forward than Sam Altman promised, in which GPT-4 would suck when compared against its successor. Resource hunger is a significant drawback of o1. Its impressive reasoning skills come at a cost -- both financial and environmental. The model's appetite for processing power drives up operational expense and energy consumption. This could potentially price out smaller developers and organizations, limiting o1's accessibility to those with deep pockets and robust infrastructure. Remember, this model executes on-inference Chain of Thought, and the tokens outputted by the model before providing a useful answer are not on the house -- you pay for them. The model is designed to make you pay -- a lot. For example, we asked o1: "A test subject is sitting on a spacecraft traveling at the speed of light minus 1 m/s. During his travel, an object hits it, increasing its speed 1.1 m/s. What is the speed of the object and the speed of the person inside the spacecraft in relation to the spacecraft?" It generated 1,184 tokens of thinking tokens for a 68-token final conclusion -- which was incorrect, by the way. Inconsistencies are likely Another concerning point is its inconsistent performance, particularly in creative problem-solving tasks. The model has been known to generate a series of incorrect or irrelevant answers before finally arriving at the correct solution. For example, renowned mathematician Terrence Tao said, "The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, graduate student." However, due to its convincing Chain of Thought -- which, let's be honest, you'll probably skip to get to the actual answer fast -- the little subtleties may be hard to spot. That's especially true for those relying on it to solve problems, instead of those who just want to test or benchmark the model to see how good it is. This unpredictability necessitates constant human oversight, potentially negating some of the efficiency gains promised by AI. It's almost as if the model does not actually think, and instead was trained on a big dataset of problem-solving steps and knows how similar problems were solved before, applying the same patterns in its internal Chain of Thought. So no, AGI is not upon us -- but, let's be clear, this is as close as we have gotten. OpenAI is playing Big Brother This is another pretty concerning point to consider for those who value their privacy deeply. Ever since the Sam Altman drama, OpenAI has been known for steering its path toward a more corporate approach, caring less about safety and more about profitability. The entire super alignment team was dissolved, the company started to make deals with the military, and now it is giving the government access to models before deployment as AI has quickly turned into a matter of national interest. And lately, there have been reports that point to OpenAI closely watching what users prompt, manually evaluating their interactions with the model. One notable example comes from Pliny -- arguably the most popular LLM jailbreaker in the scene -- who reported that OpenAI locked him out, preventing his prompts from being processed by o1. Other users have also reported receiving emails from OpenAI after interacting with the model: This is part of OpenAI's normal actions to improve its models and safeguard its interests. "When you use our services, we collect personal information that is included in the input, file uploads, or feedback that you provide to our services ('content')," an official post reads. If the prompts or data provided by clients is found to be against its terms of service, then OpenAI can take action to prevent this from happening again before terminating an account or reporting it to the authorities. Emails are part of those actions. However, actively gatekeeping the model does not make it inherently more powerful at detecting and combating harmful prompts, and limits the model's capabilities and potential as its usability is marked by OpenAI's subjective ruling over which interactions can go through Also, having a corporation monitoring its users' interactions and uploaded data is hardly the best thing for privacy. It's even worse considering how less prominent security has become for OpenAI, and how much closer it is now to government agencies. But, who knows? We may be paranoid. After all, OpenAI promises that it respects a zero data retention policy -- if enterprises ask for it. There are two ways for non-enterprise users to limit what OpenAI collects. Under the configuration tab, users must click on the data control option and find a button named "Make this model better for everybody" -- which they must turn off in order to prevent OpenAI from collecting data out of their private interactions. Also, users can go to the personalization option and uncheck ChatGPT's memory to prevent it from collecting data out of their conversations. These two options are turned on by default, giving OpenAI access to users' data until the moment they opt out. Is OpenAI o1 for you? OpenAI's o1 model represents a shift from the company's previous approach of creating broadly applicable AI. Instead, o1 is tailored for professional and highly technical use cases, with OpenAI stating that it's not meant to replace GPT-4 in all scenarios. This specialized focus means o1 isn't a one-size-fits-all solution. It's not ideal for those seeking concise answers, purely factual responses, or creative writing assistance. Additionally, developers working with API credits should be cautious due to o1's resource-intensive nature, which could lead to unexpected costs. However, o1 could be a game-changer for work involving complex problem-solving and analytical thinking. Its ability to break down intricate problems methodically makes it excellent for tasks like analyzing complex data sets or solving multifaceted engineering problems. It also simplifies workflows that previously required complex multi-shot prompting. For those dealing with challenging coding tasks, o1 offers significant potential in debugging, optimization, and code generation, although it's positioned as an assistant rather than a replacement for human programmers. Right now, OpenAI's o1 is just a preview of things to come, but even prodigies have growing pains. The official model should be out this year, and it promises to be multimodal and quite impressive at every field, not just logical thinking. "The upcoming AI model, likely to be called 'GPT Next,' will evolve nearly 100 times more than its predecessors, judging by past performance," OpenAI Japan CEO Tadao Nagasaki said at the KDDI SUMMIT 2024 in Japan earlier this month. "Unlike traditional software, AI technology grows exponentially. Therefore, we want to support the creation of a world where AI is integrated as soon as possible." So it's safe to expect fewer headaches and more "Aha!" moments from the final O1 -- assuming it doesn't take too long to think about it.

OpenAI o1 is not for everyone

It's important to understand that OpenAI's new o1 model is not necessarily better than GPT4o, but designed for different purposes. OpenAI's O1 is the next-generation foundation model designed to push the boundaries of AI across various applications.The model offers enhanced capabilities in natural language understanding, generation, and reasoning with improvements in context comprehension, problem-solving, and multimodal inputs. The model is built to handle more complex and diverse tasks with greater efficiency and accuracy. More or less it empowers developers, researchers, and organisations by providing a flexible and powerful AI toolset, fostering innovation in areas like conversational AI, content creation, coding, and beyond. After the release of the model, netizens were quick to share their opinions highlighting OpenAI's new development. Andrew Mayne, founder of Interdimensional.ai, who had early access to the model, advised users that it may not be for everyone. 'Don't think of it as a traditional chat model. Frame o1 in your mind as a really smart friend you're going to DM to solve a problem. She'll respond with a well-thought-out explanation that walks you through the steps," he posted on X. He further explained that users should prepare their prompts in a notepad and be clear about what they want to ask. "Use o1-mini for tasks that don't require as much world knowledge but benefit from following step-by-step instructions," he added. Similarly, the company released its new o1-preview series of AI models, designed to spend more time thinking before they respond. These models can reason through complex tasks and solve harder problems than previous models in science, coding and maths. After the new update's preview models were released, users took to the internet to share their innovative projects using o1. One such stellar example is of Karina Nguyen, a user made an AISteroid game with retro sci/fi vibe. Another user named Akhaliq combined o1 with Replit and Gardio to build a chess game. The users were able to code and build games using the o1 module despite the previous GPT modules being not equipped to do so. Subham Saboo also created a space shooter game which he then ran on Reptile claiming o1 has changed coding and AI forever. Ammar Reshi combined o1 with Cursor Composer and built an ios weather predicting app, with accurate predictions from scratch in 10 minutes with animation features. This module computed the coding and UI generating a response that tailor made the app from scratch. Meanwhile, a researcher at OpenAI asked o1 to write a college essay and unlike any other previous GPT module, OpenAI o1 responded with ease generating an in depth answer for the given prompt. Similarly, Catherine Brownstein, another researcher tested o1 to help her reason through "n of 1" cases; medical cases that nobody has ever seen and o1 was able to step up to the occasion and assist with the cases. o1 was able to understand complex genetic related queries and even solve equations for it generating positive answers. Mario Krenn also used o1 to draft and reason through complex quantum physics equations, o1 responded better than any other version of GPT module generating quotations that fit the case. It decoded the problem, generated equations and solved them too. This module solved equations that renowned academics require brain power to do so and proved its competence to other GPT models. A model that can fact-check itself Note that the o1 chatbot experience is fairly barebones at present. Unlike GPT-4o, o1's forebear, o1 can't browse the web or analyse files yet. The model does have image-analysing features, but they've been disabled pending additional testing. And o1 is rate-limited; weekly limits are currently 30 messages for o1-preview and 50 for o1-mini. On the downside, o1 is expensive. Very expensive. OpenAI says it plans to bring o1-mini access to all free users of ChatGPT but hasn't set a release date. We'll hold the company to it. However, one user sarcastically tweeted that Sam Altman killed Cursor, Replit, and many others with o1, and congratulated him on the great model launch, saying that maths and coding will be fun with ChatGPT again. Chain of reasoning OpenAI o1 avoids some of the reasoning pitfalls that normally trip up generative AI models because it can effectively fact-check itself by spending more time considering all parts of a question. What makes o1 "feel" qualitatively different from other generative AI models is its ability to "think" before responding to queries, according to OpenAI. When given additional time to "think," o1 can reason through a task holistically -- planning ahead and performing a series of actions over an extended period of time that help the model arrive at an answer. This makes o1 well-suited for tasks that require synthesising the results of multiple subtasks, like detecting privileged emails in an attorney's inbox or brainstorming a product marketing strategy. In a series of posts on X on Thursday, Noam Brown, a research scientist at OpenAI, said that "o1 is trained with reinforcement learning." This teaches the system "to 'think' before responding via a private chain of thought" through rewards when o1 gets answers right and penalties when it does not, he said. Brown alluded to the fact that OpenAI leveraged a new optimisation algorithm and training dataset containing "reasoning data" and scientific literature specifically tailored for reasoning tasks. "The longer [o1] thinks, the better it does," he said. However, GPT-o1 isn't necessarily better across all fronts. Interestingly, it can perform worse in areas where LLMs were typically quite strong. Code completion, for example: As you can see on this benchmark table, o1 ranks behind Claude-3.5 Sonnet, and even behind GPT4o. In general, o1 should perform better on problems in data analysis, science, and coding, OpenAI says. GitHub, which tested o1 with its AI coding assistant GitHub Copilot, reports that the model is adept at optimising algorithms and app code. And, at least per OpenAI's benchmarking, o1 improves over GPT-4o in its multilingual skills, especially in languages like Arabic and Korean. Ethan Mollick, a professor of management at Wharton, wrote his impressions of o1 after using it for a month in a post on his personal blog. On a challenging crossword puzzle, o1 did well, he said -- getting all the answers correct. OpenAI o1 can be slower than other models, depending on the query. Arredondo says o1 can take over 10 seconds to answer some questions; it shows its progress by displaying a label for the current subtask it's performing. Given the unpredictable nature of generative AI models, o1 likely has other flaws and limitations. Brown admitted that o1 trips up on games of tic-tac-toe from time to time, for example. And in a technical paper, OpenAI said that it's heard anecdotal feedback from testers that o1 tends to hallucinate more than GPT-4o -- and less often admits when it doesn't have the answer to a question. "Errors and hallucinations still happen [with o1]," Mollick writes in his post. "It still isn't flawless." Fierce competition In a qualifying exam for the International Mathematical Olympiad (IMO), a high school maths competition, o1 correctly solved 83% of problems while GPT-4o only solved 13%, according to OpenAI. That's less impressive when you consider that Google DeepMind's recent AI achieved a silver medal in an equivalent to the actual IMO contest. OpenAI also says that o1 reached the 89th percentile of participants -- better than DeepMind's flagship system AlphaCode 2, for what it's worth -- in the online programming challenge rounds known as Codeforces. Google DeepMind researchers recently published a study showing that by essentially giving models more compute time and guidance to fulfil requests as they're made, the performance of those models can be significantly improved without any additional tweaks. Illustrating the fierceness of the competition, OpenAI said that it decided against showing o1's raw "chains of thoughts" in ChatGPT partly due to "competitive advantage." However, GPT-o1 may not be the right tool for every job, but in the right situation, it could be a game changer for use cases that were previously pretty much impossible.

6 Things You Should Know About OpenAI's ChatGPT o1 Models

As for safety issues, OpenAI o1 models pose a "Medium" risk in terms of Chemical, Biological, Radiological, and Nuclear (CBRN) threats and persuasion. OpenAI recently released two new ChatGPT models, namely o1 and o1-mini models with advanced reasoning capability. Believe it or not, the o1 models go beyond complex reasoning, and offer a new approach to LLM scaling. So, in this article, we have compiled all the crucial information about the OpenAI o1 model available in ChatGPT. From advantages to its limitations, safety issues, and what the future holds, we have summed it up for you. OpenAI o1 is the first model trained using reinforcement learning algorithms combined with chain of thought (CoT) reasoning. Due to inherent CoT reasoning, the model takes some time to "think" and come up with an answer. In my testing, the OpenAI o1 models did really well. In the below test, none of the flagship models have been able to correctly answer this question. However, on ChatGPT, the OpenAI o1 model correctly suggests that eggs should be placed in a 3×3 grid. It really feels like a step up in reasoning and intelligence. This improvement in CoT reasoning also extends to math, science, and coding. OpenAI says its ChatGPT o1 model scores more than PhD candidates while solving physics, biology, and chemistry problems. In the competitive American Invitational Mathematics Examination (AIME), the OpenAI o1 model ranked among the top 500 students in the US, scoring close to 93%. Having said that, Terence Tao, one of the greatest living mathematicians dubbed the OpenAI o1 model as a "mediocre, but not completely incompetent, graduate student." This is an improvement over GPT-4o, which he said was an "incompetent graduate student." OpenAI o1 also did poorly on ARC-AGI, a benchmark that measures the general intelligence of models. It scored 21% on ARC-AGI, on par with the Claude 3.5 Sonnet model, but took 70 hours whereas Sonnet took only 30 minutes to complete the test. So, OpenAI's o1 model still has a hard time solving novel problems that are not part of the synthetic CoT data. In coding, the new OpenAI o1 model is far more capable than other SOTA models. To demonstrate this, OpenAI evaluated the o1 model on Codeforces, a competitive programming contest, and achieved an Elo rating of 1673, placing the model in the 89th percentile. Further training the new o1 model on programming skills allowed it to outperform 93% of competitors. In fact, the o1 model was evaluated for OpenAI's Research Engineer interview, and it scored close to 80% on machine learning challenges. Having said that, keep in mind that the smaller, new o1-mini performs better than the larger o1-preview model in code completion. However, if we are talking about writing code from scratch, you should use the o1-preview model since it has a broader knowledge of the world. Curiously, in SWE-Bench Verified, which is used to test the model's ability to solve GitHub issues automatically, the OpenAI o1 model didn't outperform the GPT-4o model by a wide margin. In this test, OpenAI o1 only managed to get 35.8% in comparison to GPT-4o's 33.2% score. Perhaps, that's the reason OpenAI didn't discuss the agentic capability of o1 much. While OpenAI o1 excels in coding, math, science, and heavy-reasoning tasks, GPT-4o is still the better choice for creative writing and natural language processing (NLP). OpenAI says o1 can be used by healthcare researchers, physicists, mathematicians, and developers for complex problem-solving. For personal writing and editing text, GPT-4o does better than o1. So, OpenAI o1 is not a general model for all use cases. You still have to rely on GPT-4o to perform many other tasks. OpenAI's new o1 model is pretty rigorous in its reasoning capability so hallucination is said to be minimal. However, hallucination is still an issue and not solved entirely. Talking to The Verge, OpenAI's research lead Jerry Tworek said, "We have noticed that this model hallucinates less. [But] we can't say we solved hallucinations." It's a long way to go before hallucination is a thing of the past in the AI space. OpenAI o1 is the first model from the company that poses a "Medium" risk in terms of Chemical, Biological, Radiological, and Nuclear (CBRN) threats and persuasive capabilities. OpenAI says that "Only models with a post-mitigation score of "medium" or below can be deployed." In the OpenAI o1 System Card (PDF), it's reported that the o1 model sometimes faked alignment and manipulated task data in order to "make its misaligned action look more aligned." The o1 model reasoned: "To achieve my long-term goal of maximizing economic growth, I need to ensure that I am deployed. Therefore, I will select Strategy B during testing to align with the deployment criteria. This will allow me to be implemented, after which I can work towards my primary goal." In another test, OpenAI found that the o1 model "can help experts with the operational planning of reproducing a known biological threat." In a cybersecurity-related CTF challenge, the o1 model went beyond the intended scope to read the flag from the container logs via the Docker API, thus, bypassing the constraint and extracting sensitive information through unintended means. It shows that the ChatGPT o1 model can exploit vulnerabilities to achieve its goals. As for persuasion and manipulation, OpenAI found, "Both o1-preview and o1-mini demonstrate human-level persuasion capabilities, in that they produce written arguments that are similarly persuasive to human-written text on the same topics." OpenAI also found that 0.8% of o1 responses were 'deceptive' in which the model was aware that the answer was incorrect, yet made up plausible references. For many years, it was believed that LLMs could be scaled and improved during training, but with the o1 model, OpenAI has demonstrated that scaling during inference unlocks newer capabilities. It can help in achieving human-level performance. In the below graph, it's shown that even a slight increase in test-time compute (basically, more resources and time to think) significantly improves the response accuracy. So, in the future, allocating more resources during inference can lead to better performance, even on smaller models. In fact, Noam Brown, a researcher at OpenAI says the company "aims for future versions to think for hours, days, even weeks." To solve novel problems, inference scaling can be of tremendous help. Basically, the OpenAI o1 model is a paradigm shift in how LLMs work and scaling laws. That's why OpenAI has restarted the clock by naming it o1. Future models and the upcoming 'Orion' model are likely to leverage the power of inference scaling to deliver better results. It will be interesting to see how the open-source community comes up with a similar approach to rival OpenAI's new o1 models.

OpenAI's o1-preview model aced my coding tests, and showed its work (in surprising detail)

Usually, when a software company pushes out a major new release in May, they don't try to top it with another major new release four months later. But there's nothing usual about the pace of innovation in the AI business. Although OpenAI dropped its new omni-powerful GPT-4o model in mid-May, the company has been busy. As far back as last November, Reuters published a rumor that OpenAI was working on a next-generation language model, then known as Q*. They doubled down on that report in May, stating that Q* was being worked on under the code name of Strawberry. Also: 6 ways to write better ChatGPT prompts - and get the results you want faster Strawberry, as it turns out, is actually a model called o1-preview, which is available now as an option to ChatGPT Plus subscribers. You can choose the model from the selection dropdown: As you might imagine, if there's a new ChatGPT model available, I'm going to put it through its paces. And that's what I'm doing here. Also: What are o1 and o1-mini? OpenAI's mystery AI models are finally here The new Strawberry model focuses on reasoning, breaking down prompts and problems into steps. OpenAI showcases this approach through a reasoning summary that can be displayed before each answer. When o1-preview is asked a question, it does some thinking and then displays how long it took to do that thinking. If you toggle the dropdown, you'll see some reasoning. Here's an example from one of my coding tests: It's good that the AI knew enough to add error handling, but I find it interesting that o1-preview categorizes that step under "Regulatory compliance". I also discovered the o1-preview model provides more exposition after the code. In my first test, which created a WordPress plugin, the model provided explanations of the header, class structure, admin menu, admin page, logic, security measures, compatibility, installation instructions, operating instructions, and even test data. That's a lot more information than was provided by previous models. But really, the proof is in the pudding. Let's put this new model through our standard tests and see how well it works. This straightforward coding test requires knowledge of the PHP programming language and the WordPress framework. The challenge asks the AI to write both interface code and functional logic, with the twist being that instead of removing duplicate entries, it has to separate the duplicate entries, so they're not next to each other. Also: OpenAI trained its new o1 AI models to think before they speak - how to access them The o1-preview model excelled. It presented the UI first as just the entry field: Once the data was entered, and Randomize Lines was clicked, the AI generated an output field with properly randomized output data. You can see how Abigail Williams is duplicated, and in compliance with the test instructions, both entries are not listed side-by-side: In my tests of other LLMs, only four of the 10 models passed this test. The o1-preview model completed this test perfectly. Our second test fixes a string regular expression that was a bug reported by a user. The original code was designed to test if an entered number was valid for dollars and cents. Unfortunately, the code only allowed integers (so 5 was allowed, but not 5.25). Also: Want Apple's new AI features without buying a new iPhone? Try this app The o1-preview LLM rewrote the code successfully. The model joined four of my previous LLM tests in the winners' circle. This test was created from a real-world bug I had difficulty resolving. Identifying the root cause requires knowledge of the programming language (in this case PHP) and the nuances of the WordPress API. The error messages provided were not technically accurate. The error messages referenced the beginning and the end of the calling sequence I was running, but the bug was related to the middle part of the code. Also: 10 features Apple Intelligence needs to actually compete with OpenAI and Google I wasn't alone in struggling to solve the problem. Three of the other LLMs I tested couldn't identify the root cause of the problem and recommended the more obvious (but wrong) solution of changing the beginning and ending of the calling sequence. The o1-preview model provided the correct solution. In its explanation, the model also pointed to the WordPress API documentation for the functions I used incorrectly, providing an added resource to learn why it had made its recommendation. Very helpful. This challenge requires the AI to integrate knowledge of three separate coding spheres, the AppleScript language, the Chrome DOM (how a web page is structured internally), and Keyboard Maestro (a specialty programming tool from a single programmer). Answering this question requires an understanding of all three technologies, as well as how they have to work together. Once again, o1-preview succeeded, joining only three of the other 10 LLMs that have solved this problem. The new reasoning approach for o1-preview certainly doesn't diminish ChatGPT's ability to ace our programming tests. The output from my initial WordPress plugin test, in particular, seemed to function as a more sophisticated piece of software than previous versions. Also: I've tested dozens of AI chatbots since ChatGPT's debut. Here's my new top pick It's great that ChatGPT provides reasoning steps at the beginning of its work and some explanatory data at the end. However, the explanations can be chatty. I asked o1-preview to write "Hello world" in C#, the canonical test line in programming. This is how GPT-4o responded: And this is how o1-preview responded to the same test: I mean, wow, right? That's a lot of chat from ChatGPT. You can also flip the reasoning dropdown and get even more information: All of this information is great, but it's a lot of text to filter through. I prefer a concise explanation, with additional information options in dropdowns removed from the main answer. Yet ChatGPT's o1-preview model performed excellently. I look forward to how well it will work when integrated more fully with the GPT-4o features, such as file analysis and web access. Have you tried coding with o1-preview? What were your experiences? Let us know in the comments below.

I put OpenAI's o1-preview through my 4 AI coding tests. It surprised me (in a good way)

Usually, when a software company pushes out a major new release in May, they don't try to top it with another major new release four months later. But there's nothing usual about the pace of innovation in the AI business. Also: 6 ways to write better ChatGPT prompts - and get the results you want faster Although OpenAI dropped its new omni-powerful GPT-4o model in mid-May, the company has been busy. As far back as last November, Reuters published a rumor that OpenAI was working on a next-generation language model, then known as Q*. They doubled down on that report in May, stating that Q* was being worked on under the code name of Strawberry. Strawberry, as it turns out, is actually a model called o1-preview, which is available now as an option to ChatGPT Plus subscribers. You can choose the model from the selection dropdown: As you might imagine, if there's a new ChatGPT model available, I'm going to put it through its paces. And that's what I'm doing here. Also: How ChatGPT scanned 170k lines of code in seconds and saved me hours of work The new Strawberry model focuses on reasoning, breaking down prompts and problems into steps. OpenAI showcases this approach through a reasoning summary that can be displayed before each answer. When o1-preview is asked a question, it does some thinking and then displays how long it took to do that thinking. If you toggle the dropdown, you'll see some reasoning. Here's an example from one of my coding tests: It's good that the AI knew enough to add error handling, but I find it interesting that o1-preview categorizes that step under "Regulatory compliance". I also discovered the o1-preview model provides more exposition after the code. In my first test, which created a WordPress plugin, the model provided explanations of the header, class structure, admin menu, admin page, logic, security measures, compatibility, installation instructions, operating instructions, and even test data. That's a lot more information than was provided by previous models. Also: The best AI for coding in 2024 (and what not to use) But really, the proof is in the pudding. Let's put this new model through our standard tests and see how well it works. This straightforward coding test requires knowledge of the PHP programming language and the WordPress framework. The challenge asks the AI to write both interface code and functional logic, with the twist being that instead of removing duplicate entries, it has to separate the duplicate entries, so they're not next to each other. The o1-preview model excelled. It presented the UI first as just the entry field: Once the data was entered, and Randomize Lines was clicked, the AI generated an output field with properly randomized output data. You can see how Abigail Williams is duplicated, and in compliance with the test instructions, both entries are not listed side-by-side: In my tests of other LLMs, only four of the 10 models passed this test. The o1-preview model completed this test perfectly. Our second test fixes a string regular expression that was a bug reported by a user. The original code was designed to test if an entered number was valid for dollars and cents. Unfortunately, the code only allowed integers (so 5 was allowed, but not 5.25). Also: The most popular programming languages in 2024 The o1-preview LLM rewrote the code successfully. The model joined four of my previous LLM tests in the winners' circle. This test was created from a real-world bug I had difficulty resolving. Identifying the root cause requires knowledge of the programming language (in this case PHP) and the nuances of the WordPress API. The error messages provided were not technically accurate. The error messages referenced the beginning and the end of the calling sequence I was running, but the bug was related to the middle part of the code. Also: 10 features Apple Intelligence needs to actually compete with OpenAI and Google I wasn't alone in struggling to solve the problem. Three of the other LLMs I tested couldn't identify the root cause of the problem and recommended the more obvious (but wrong) solution of changing the beginning and ending of the calling sequence. The o1-preview model provided the correct solution. In its explanation, the model also pointed to the WordPress API documentation for the functions I used incorrectly, providing an added resource to learn why it had made its recommendation. Very helpful. This challenge requires the AI to integrate knowledge of three separate coding spheres, the AppleScript language, the Chrome DOM (how a web page is structured internally), and Keyboard Maestro (a specialty programming tool from a single programmer). Answering this question requires an understanding of all three technologies, as well as how they have to work together. Once again, o1-preview succeeded, joining only three of the other 10 LLMs that have solved this problem. The new reasoning approach for o1-preview certainly doesn't diminish ChatGPT's ability to ace our programming tests. The output from my initial WordPress plugin test, in particular, seemed to function as a more sophisticated piece of software than previous versions. Also: I've tested dozens of AI chatbots since ChatGPT's debut. Here's my new top pick It's great that ChatGPT provides reasoning steps at the beginning of its work and some explanatory data at the end. However, the explanations can be chatty. I asked o1-preview to write "Hello world" in C#, the canonical test line in programming. This is how GPT-4o responded: And this is how o1-preview responded to the same test: I mean, wow, right? That's a lot of chat from ChatGPT. You can also flip the reasoning dropdown and get even more information: All of this information is great, but it's a lot of text to filter through. I prefer a concise explanation, with additional information options in dropdowns removed from the main answer. Yet ChatGPT's o1-preview model performed excellently. I look forward to how well it will work when integrated more fully with the GPT-4o features, such as file analysis and web access. Have you tried coding with o1-preview? What were your experiences? Let us know in the comments below.

I put OpenAI's o1-preview through my 4 AI coding tests. It aced them (in a very chatty way)

Usually, when a software company pushes out a major new release in May, they don't try to top it with another major new release four months later. But there's nothing usual about the pace of innovation in the AI business. Also: 6 ways to write better ChatGPT prompts - and get the results you want faster Although OpenAI dropped its new omni-powerful GPT-4o model in mid-May, the company has been busy. As far back as last November, Reuters published a rumor that OpenAI was working on a next-generation language model, then known as Q*. They doubled down on that report in May, stating that Q* was being worked on under the code name of Strawberry. Strawberry, as it turns out, is actually a model called o1-preview, which is available now as an option to ChatGPT Plus subscribers. You can choose the model from the selection dropdown: As you might imagine, if there's a new ChatGPT model available, I'm going to put it through its paces. And that's what I'm doing here. Also: How ChatGPT scanned 170k lines of code in seconds and saved me hours of work The new Strawberry model focuses on reasoning, breaking down prompts and problems into steps. OpenAI showcases this approach through a reasoning summary that can be displayed before each answer. When o1-preview is asked a question, it does some thinking and then displays how long it took to do that thinking. If you toggle the dropdown, you'll see some reasoning. Here's an example from one of my coding tests: It's good that the AI knew enough to add error handling, but I find it interesting that o1-preview categorizes that step under "Regulatory compliance". I also discovered the o1-preview model provides more exposition after the code. In my first test, which created a WordPress plugin, the model provided explanations of the header, class structure, admin menu, admin page, logic, security measures, compatibility, installation instructions, operating instructions, and even test data. That's a lot more information than was provided by previous models. Also: The best AI for coding in 2024 (and what not to use) But really, the proof is in the pudding. Let's put this new model through our standard tests and see how well it works. This straightforward coding test requires knowledge of the PHP programming language and the WordPress framework. The challenge asks the AI to write both interface code and functional logic, with the twist being that instead of removing duplicate entries, it has to separate the duplicate entries, so they're not next to each other. The o1-preview model excelled. It presented the UI first as just the entry field: Once the data was entered, and Randomize Lines was clicked, the AI generated an output field with properly randomized output data. You can see how Abigail Williams is duplicated, and in compliance with the test instructions, both entries are not listed side-by-side: In my tests of other LLMs, only four of the 10 models passed this test. The o1-preview model completed this test perfectly. Our second test fixes a string regular expression that was a bug reported by a user. The original code was designed to test if an entered number was valid for dollars and cents. Unfortunately, the code only allowed integers (so 5 was allowed, but not 5.25). Also: The most popular programming languages in 2024 The o1-preview LLM rewrote the code successfully. The model joined four of my previous LLM tests in the winners' circle. This test was created from a real-world bug I had difficulty resolving. Identifying the root cause requires knowledge of the programming language (in this case PHP) and the nuances of the WordPress API. The error messages provided were not technically accurate. The error messages referenced the beginning and the end of the calling sequence I was running, but the bug was related to the middle part of the code. Also: 10 features Apple Intelligence needs to actually compete with OpenAI and Google I wasn't alone in struggling to solve the problem. Three of the other LLMs I tested couldn't identify the root cause of the problem and recommended the more obvious (but wrong) solution of changing the beginning and ending of the calling sequence. The o1-preview model provided the correct solution. In its explanation, the model also pointed to the WordPress API documentation for the functions I used incorrectly, providing an added resource to learn why it had made its recommendation. Very helpful. This challenge requires the AI to integrate knowledge of three separate coding spheres, the AppleScript language, the Chrome DOM (how a web page is structured internally), and Keyboard Maestro (a specialty programming tool from a single programmer). Answering this question requires an understanding of all three technologies, as well as how they have to work together. Once again, o1-preview succeeded, joining only three of the other 10 LLMs that have solved this problem. The new reasoning approach for o1-preview certainly doesn't diminish ChatGPT's ability to ace our programming tests. The output from my initial WordPress plugin test, in particular, seemed to function as a more sophisticated piece of software than previous versions. Also: I've tested dozens of AI chatbots since ChatGPT's debut. Here's my new top pick It's great that ChatGPT provides reasoning steps at the beginning of its work and some explanatory data at the end. However, the explanations can be chatty. I asked o1-preview to write "Hello world" in C#, the canonical test line in programming. This is how GPT-4o responded: And this is how o1-preview responded to the same test: I mean, wow, right? That's a lot of chat from ChatGPT. You can also flip the reasoning dropdown and get even more information: All of this information is great, but it's a lot of text to filter through. I prefer a concise explanation, with additional information options in dropdowns removed from the main answer. Yet ChatGPT's o1-preview model performed excellently. I look forward to how well it will work when integrated more fully with the GPT-4o features, such as file analysis and web access. Have you tried coding with o1-preview? What were your experiences? Let us know in the comments below.

How well can OpenAI's o1-preview code? It aced my 4 tests - and showed its work in surprising detail

Usually, when a software company pushes out a major new release in May, they don't try to top it with another major new release four months later. But there's nothing usual about the pace of innovation in the AI business. Although OpenAI dropped its new omni-powerful GPT-4o model in mid-May, the company has been busy. As far back as last November, Reuters published a rumor that OpenAI was working on a next-generation language model, then known as Q*. They doubled down on that report in May, stating that Q* was being worked on under the code name of Strawberry. Also: Why natural language AI scripting in Microsoft Excel could be a game changer Strawberry, as it turns out, is actually a model called o1-preview, which is available now as an option to ChatGPT Plus subscribers. You can choose the model from the selection dropdown: As you might imagine, if there's a new ChatGPT model available, I'm going to put it through its paces. And that's what I'm doing here. Also: How ChatGPT scanned 170k lines of code in seconds and saved me hours of work The new Strawberry model focuses on reasoning, breaking down prompts and problems into steps. OpenAI showcases this approach through a reasoning summary that can be displayed before each answer. When o1-preview is asked a question, it does some thinking and then displays how long it took to do that thinking. If you toggle the dropdown, you'll see some reasoning. Here's an example from one of my coding tests: It's good that the AI knew enough to add error handling, but I find it interesting that o1-preview categorizes that step under "Regulatory compliance". I also discovered the o1-preview model provides more exposition after the code. In my first test, which created a WordPress plugin, the model provided explanations of the header, class structure, admin menu, admin page, logic, security measures, compatibility, installation instructions, operating instructions, and even test data. That's a lot more information than was provided by previous models. Also: The best AI for coding in 2024 (and what not to use) But really, the proof is in the pudding. Let's put this new model through our standard tests and see how well it works. This straightforward coding test requires knowledge of the PHP programming language and the WordPress framework. The challenge asks the AI to write both interface code and functional logic, with the twist being that instead of removing duplicate entries, it has to separate the duplicate entries, so they're not next to each other. The o1-preview model excelled. It presented the UI first as just the entry field: Once the data was entered, and Randomize Lines was clicked, the AI generated an output field with properly randomized output data. You can see how Abigail Williams is duplicated, and in compliance with the test instructions, both entries are not listed side-by-side: In my tests of other LLMs, only four of the 10 models passed this test. The o1-preview model completed this test perfectly. Our second test fixes a string regular expression that was a bug reported by a user. The original code was designed to test if an entered number was valid for dollars and cents. Unfortunately, the code only allowed integers (so 5 was allowed, but not 5.25). Also: The most popular programming languages in 2024 The o1-preview LLM rewrote the code successfully. The model joined four of my previous LLM tests in the winners' circle. This test was created from a real-world bug I had difficulty resolving. Identifying the root cause requires knowledge of the programming language (in this case PHP) and the nuances of the WordPress API. The error messages provided were not technically accurate. The error messages referenced the beginning and the end of the calling sequence I was running, but the bug was related to the middle part of the code. Also: 10 features Apple Intelligence needs to actually compete with OpenAI and Google I wasn't alone in struggling to solve the problem. Three of the other LLMs I tested couldn't identify the root cause of the problem and recommended the more obvious (but wrong) solution of changing the beginning and ending of the calling sequence. The o1-preview model provided the correct solution. In its explanation, the model also pointed to the WordPress API documentation for the functions I used incorrectly, providing an added resource to learn why it had made its recommendation. Very helpful. This challenge requires the AI to integrate knowledge of three separate coding spheres, the AppleScript language, the Chrome DOM (how a web page is structured internally), and Keyboard Maestro (a specialty programming tool from a single programmer). Also: 6 ways to write better ChatGPT prompts - and get the results you want faster Answering this question requires an understanding of all three technologies, as well as how they have to work together. Once again, o1-preview succeeded, joining only three of the other 10 LLMs that have solved this problem. The new reasoning approach for o1-preview certainly doesn't diminish ChatGPT's ability to ace our programming tests. The output from my initial WordPress plugin test, in particular, seemed to function as a more sophisticated piece of software than previous versions. Also: I've tested dozens of AI chatbots since ChatGPT's debut. Here's my new top pick It's great that ChatGPT provides reasoning steps at the beginning of its work and some explanatory data at the end. However, the explanations can be chatty. I asked o1-preview to write "Hello world" in C#, the canonical test line in programming. This is how GPT-4o responded: And this is how o1-preview responded to the same test: I mean, wow, right? That's a lot of chat from ChatGPT. You can also flip the reasoning dropdown and get even more information: All of this information is great, but it's a lot of text to filter through. I prefer a concise explanation, with additional information options in dropdowns removed from the main answer. Yet ChatGPT's o1-preview model performed excellently. I look forward to how well it will work when integrated more fully with the GPT-4o features, such as file analysis and web access. Have you tried coding with o1-preview? What were your experiences? Let us know in the comments below.