Image Credit - John DaSilva @ Action Items Lab
Which platform is the best at conducting market research, Open AI’s ChatGPT or Google's Bard?
AI-driven tools like ChatGPT and Bard are revolutionizing the way we conduct market research. In this experiment, I've put these two platforms to the test in a real-life scenario involving hotel investment research. Spoiler alert, Bard came out the winner.
Read on to find out how I conducted the experiment, why Bard won, and where Bard and
ChatGPT need improvement.
How the Review Was Conducted
The review process involved a step-by-step comparison of ChatGPT and Bard across various stages of market research. Each platform was given identical tasks and information. The evaluation criteria included ease of use, document handling capabilities, depth and relevance of analysis, ability to retain context, and the quality of insights provided. The platforms were assessed on their ability to process the same set of data and instructions, ensuring a fair and unbiased comparison.
Setting the Scene: The Initial Research
Before diving into the live experiment, it's crucial to note the groundwork. I had already conducted preliminary market research, examining global and local trends in the hospitality sector. The focus of this live test was to delve deeper, particularly into local trends and competitor analysis, to pinpoint the right market segment and pricing strategy.
The Experiment Begins: Establishing Context with Custom Instructions in ChatGPT and Bard (Memory)
To ensure fairness, I provided both ChatGPT and Bard with the same set of custom instructions I've refined for such analyses. In ChatGPT, this meant inputting a set of custom instructions I created and refined to do this kind of work. Google is working on a feature call “Memory,” which will be similar to ChatGPT’s custom instructions.
For now, each conversation with Bard starts with a clean slate, though Google Bard claims to retain some snippets of information. As such, I manually input my custom instructions into Bard to be sure it was working from a level playing field.
Aside from having to remember to reenter this if I start a new conversation in Bard, I am concerned Bard applies these instructions to the character limits on how much information it can remember and draw from in the conversation, making it less useful. Point ChatGPT.
Document Analysis: A Challenge for Bard
A key part of my research involved analyzing existing documents. ChatGPT's file upload capability shone here, effortlessly handling my documents. Bard, lacking this feature, required manual text inputs, a noticeable drawback, especially with complex PDFs and large spreadsheets. To work around this, I had to ask ChatGPT to take a PDF and change it to text so I could paste it into Bard (simply converting would have resulted in a very messy document, as there were many tables, including a Business Canvas). Point ChatGPT.
Analysis and Insights: A Surprising Turn from Bard
Once the information was fed into the platforms, each conducted and presented some basic level analysis. ChatGPT provided better information on the document itself. Bard went beyond the given data and seemed to better understand what I was trying to accomplish. Remarkably, it created an almost complete SWOT analysis, although it wasn't explicitly present in my documents. This proactive approach was a pleasant surprise. Point Bard.
Market Segmentation: Diverse Approaches for Both Bard and ChatGPT
I pasted four detailed market segments I created that included detailed customer needs, preferences, and motivations into both platforms. I then asked each platform to analyze my work and make any suggestions.
ChatGPT's response was a precise summary with specific considerations. Bard, however, provided broader insights, unexpectedly highlighting the crucial aspect of property location, a point I wholeheartedly agreed with. Point Bard.
Investment Analysis: The Local Context
Next, I asked each to help me conduct an investment analysis on the segments and redo them with a local focus. Both did a good job of providing top level analysis on pricing, ROI and risk. I did like the way Bard presented the information better. Bard also provided some additional advice and considerations that ChatGPT did not provide, though none of this was something I had not already considered. Half a point Bard.
Narrowing Down: Wise Choices Affirmed by both ChatGPT and Bard
For my next step, I ruled out two market segments that would not be pursued due to cost, competition, or risk. After deciding to eliminate two market segments, both platforms affirmed my decision. Bard offered general advice, helpful for those less familiar with the industry. No clear winner.
In-Depth Queries: Surprising Insights from Bard
I then asked a series of more detailed questions, just to make sure I exhausted this topic.
Bard generated some surprising new insights, prompting me to reconsider my ultimate goals and values. However, Bard needed reminders about the research context as I had gone beyond its memory capability, a point where ChatGPT showed more consistency. Slight edge to Bard.
Property Categorization and Commonalities: Bard's Strength
Next, I provided a list of properties that fell within each of the two segments I selected for further analysis, and asked each platform to categorize the properties into the best segment. There were a lot of properties here, and each came up with slightly different response, but nothing contradictory. I even cross refenced the lists in each platform and neither platform disagreed with the other. I then asked them to find commonalities across these properties. Here Bard excelled in identifying commonalities across properties, a crucial aspect of my analysis. Point Bard.
Pricing and Packages: Bard Takes the Lead
I then set about doing a price comparison analysis. Here, ChatGPT stumbled badly, especially in extracting specific package prices from websites. Bard, on the other hand, started out fairly superficial. But, when asked to deep dive, it gave some very good information, breaking out price ranges and specific package offerings at each of the properties I identified. Bard really impressed me with this. Bard, however, could not create a “downloadable table” from this information, but it did create one in within the conversation thread, and then allow me to export that to Google Sheets.
This difference is likely due to their training sets and real time access to up-to-date data. At this time, ChatGPT training data goes to April 2023. While Bard is up to date. And while ChatGPT can browse the internet, its capabilities to interact with a website are limited. Point Bard.
User Experience and Presentation in Bard and ChatGPT
Bard consistently impressed with its attentiveness, frequently seeking clarification when faced with ambiguities, a stark contrast to ChatGPT, which often forged ahead with its best guess, at times leading to inefficient use of its memory capacity. Bard's presentation of information was notably polished, offering data in well-structured, easily digestible formats that translated smoothly into clean copy-paste operations – a true help for research-heavy tasks.
However, Bard wasn't without its issues; it occasionally became entangled in my requests for clarification, offering affirmations rather than concrete actions or answers. And it hallucinated some very specific tourism facts I could not verify.
Conclusion: Bard, the Market Research Champion
To sum up, Bard's ability to provide real-time data and generate novel insights makes it a superior tool for market research, especially in dynamic sectors like hotel investments. While ChatGPT's user-friendliness and memory retention are superior, Bard’s strength in delivering up-to-date and comprehensive analysis sets it apart as the preferred choice for this specific task.
Comments