Just a few days ago, OpenAI made an exciting update by introducing web browsing and Plugins to all ChatGPT Plus users. I thought it might be a good time to delve into some of the plugins I find useful and discuss their use cases in web browsing, summarizing, coding, and more.
There are at least three tools that are available for web browsing:
- OpenAI’s official Bing search
- KeyMate.AI Search
All three tools work great with basic questions. But I’d like to find an example where it takes several steps for the language model and the Web Browsing tools to find the answer. Here is the question I came up with that needs three steps of web searching:
“Who succeeded Woody Johnson as the US ambassador to the United Kingdom, and which President did she serve under? Additionally, what is the current age of this President?”
Surprisingly, all three tools got this question wrong. Well, the final answers might be okay, but the intermediate answers are wrong.
1.1 Bing search (brand new)
Microsoft just announced that they are bringing Bing to ChatGPT as the default search experience. As I was writing this blog post tonight, I just noticed I got access to Bing search on ChatGPT!
Bing search did the right search terms but it has returned the wrong answer for the US ambassador. Jane Hartley is current US Ambassador, not the successor of Woody Johnson.
There are several things I liked and disliked about this tool:
✅ Bing search: I like that we know the exact search term and we can click on the search term and see the search results.
✅ Thinking: Looks like OpenAI gave LLMs a chance to think about what to do at each step.
✅ Citations: I really like the citations in the answers.
❎ One thing I wish to have is to see more clearly what’s going on under the hood. Maybe there is a way. I just couldn’t find it. Let me know if anyone knows.
❎ We can’t use Bing search with other plugins. It’d be a lot more powerful to combine Bing’s search power with plugin capabilities. I guess right now the alternative is to use one of the search plugins like WebPilot to interact with other plugins.
1.2 OpenAI’s previous Official Web Browsing (replaced with Bing search)
OpenAI’s previous Web Browsing was able to answer this question correctly with three steps:
1.3 KeyMate.AI Search
Well, this tool just gave all kinds of wrong answers. It didn’t even return a legitimate ambassador. Also, when looking at the steps, it browsed way too many documents and it’d not clear which document it used to get the final answer.
WebPilot is an open-source tool. You can find its Github repo here. WebPilot wasn’t able to find who succeeded Woody Johnson. The correct answer should be Yael Lempert instead of Jane D. Hartley, the current ambassador.
✅ I like that it provides what’s going on under the hood: the web link, the content of the link, and rules for how the language model should respond.
✅ WebPilot can also answer with a specified tone, translate, rewrite, and more.
All of the tools above can be used to summarize web pages as well:
2.1 OpenAI Official Web Browsing
2.2 KeyMate.AI Search
2.4 Link Reader
The Link Reader tool is designed to read the content of all kinds of links, including webpage, PDF, PPT, image, Word & other docs.
VoxScript is super useful in terms of summarizing YouTube videos. I actually find this tool the most useful for me.
ChatWithVideo does a good job summarizing the video as well. I find it takes longer time when I ask a question about the video.
I don’t seem to get much luck with VideoInsights.
I can’t believe there is only one ChatGPT coding plugin available and that is Noteable. They have certainly taken the opportunity to showcase and promote their offering. While I haven’t had a chance to use Noteable before, since it is the only coding-related plugin, I have to give it a try, and the results are impressive.
Here is the prompt I used (source):
I’m a coordinator for a major art museum in the United States. I’m considering what collections to feature in the coming year. Please create and execute a notebook to analyze the data above and provide charts and graphs as well as descriptions of trends and anomalies to tell me story of how artists in the MoMA are or are not being represented based on gender, nationality and historical period. Please remember that I’m not experienced with data analysis so include text in the notebook explaining to me what you’re doing and why it’s important in a way that is accessible.
The Noteable plugin was able to direct me to create a new project and use the new project I created:
The result is a full report with data cleaning, visualization, and insights! Wow! If you look closely, there are two places where the code didn’t run and returned an error message. Noteable was able to rerun and fix the errors:
“It seems like there was an error because the ‘artists’ variable is not defined. This could be due to a kernel restart which would clear all the variables. Let’s rerun the data loading and missing value handling cells.”
More thoughts on the Noteable plugin:
✅ I love how easy it is to generate reports and the UI looks great. I wasn’t expecting a full report. That was definitely a pleasant surprise. I also found more examples with more complicated use cases including various modeling and various styles of visualizations.
❎ It took a little longer than I thought on each run and I get different results every time. I wish it could be faster and generate more consistent results. But that’s not actually an issue with Noteable, it’s more of an issue of using the OpenAI models. Everything generated is probabilistic, there is no guarantee to get the same result every time.
What’s your favorite ChatGPT plugin? And do you have a guess on what’s my favorite? It has to be the Comic Finder plugin.
You can combine up to three plugins to use. For instance, WebPilot was able to answer who is the UK prime minister and Comic Finder found some comics related to politics. The output was informative, clarifying that although these comics don’t directly mention the UK prime minister, they are related to the realm of politics. Very smart!
- Wolfram Alpha: Undoubtedly, Wolfram Alpha stands out as one of the most dependable ChatGPT plugins. It offers access to robust computation, precise mathematical capabilities, carefully curated knowledge, real-time data, and visualization tools.
- Zapier: With the ability to automate tasks across more than 5,000 app integrations, Zapier is a highly useful plugin for streamlining your workflow.
- PDF Interactions: There is a wide range of plugins available for interacting with PDF files. For instance, the previously mentioned Link Reader plugin works well with PDF files. Additionally, plugins like AskYourPDF and ChatWithPDF offer further functionality in this area.
- Finance-related Plugins: It’s worth noting the abundance of plugins focused on finance-related tasks, specifically for obtaining market data and insights.
- User Interface: The user interface could use some improvement. With “only” 128 plugins, it already feels overwhelming. It would be helpful to have a search bar for easily finding specific plugins, and some plugin descriptions lack informative details.
- Trustworthiness of Plugins: I’m uncertain about the trustworthiness of the plugins. The only plugin I trust is the Wolfram plugin because it’s developed by the renowned Wolfram Alpha. However, for the other plugins, I’m unsure about the developers behind them, and I don’t have the time to investigate each one. It would be beneficial if the plugins were hosted similarly to app store apps, with links, demos, and ratings on the app store, ensuring more transparency.
- Compatibility with Web Browsing: Currently, the official Web Browsing plugin does not work alongside other plugins. It would be helpful if it could be used with other plugins seamlessly.
- Limitations on Plugin Usage: It can be frustrating to only be able to use three plugins at a time. It would be preferable if the language model itself could determine which installed plugins to utilize, rather than restricting users to selecting only three. While I understand that more plugins may consume additional token space, having the option for more plugins would be appreciated.
- Integration of Plugins in Code: I’m unsure about how to incorporate plugins into the OpenAI API. It would be great if there was a consistent and straightforward method for interacting with plugins in Python.
The recent announcements made at the Microsoft Build conference have truly elevated the ChatGPT plugins to a whole new level. Microsoft is building an AI plugins ecosystem that works across platforms:
Developers can now use one platform to build and submit plugins that work across both consumer and business surfaces, including ChatGPT, Bing, Dynamics 365 Copilot, Microsoft 365 Copilot, and Windows Copilot.
The ChatGPT plugins are certainly still in the early stages of development. I anticipate there will be an influx of new plugins coming empowering and unlocking the potential of AI systems. I look forward to exploring more plugins and witnessing the progress in the future.
. . .
By Sophia Yang on May 24, 2023