Close to Home: Is chatbot the end of the road for sommeliers?

When I heard that ChatGPT — the new artificial intelligence-powered chatbot — scored an impressive 92% on the introductory Court of Master Sommeliers test, I decided to test its knowledge of Sonoma wines, wineries and winemaking.

I’m a vintner. And I wanted to know if ChatGPT’s artificial intelligence is ready for the tasting room, or if it’s just another overhyped technology, like the metaverse and blockchain, that still needs time to ferment.

A ton of money is riding on ChatGPT’s potential to render entire job categories, including sommeliers, obsolete. But is AI-induced job displacement still years away, or are wine stewards at risk?

I started with the only Sonoma-specific test question from Master Sommelier Fernando Betateta’s website and fed it to ChatGPT.

The question was: Name all the American Viticulture Areas in Sonoma County that end in “Valley”?

ChatGPT’s answer: Alexander Valley, Bennett Valley, Chalk Hill Valley, Knights Valley, Moon Mountain District Sonoma Valley, Sonoma Valley.

At first, I was astonished because I thought it was mostly correct. But when I examined the answer more carefully, I realized it was mostly wrong.

The Sonoma AVAs that end in “Valley” are: Alexander Valley, Bennett Valley, Dry Creek Valley, Green Valley, Knights Valley, Russian River Valley, Sonoma Valley.

ChatGPT named four of the seven local appellations ending in “Valley” (57% accuracy) and added “Valley” to the end of Moon Mountain District and Chalk. It left out entirely the Russian River Valley, which has become the most desirable appellation for pinot noir (and one we make).

Next, I asked, “What is the difference in size between Napa and Sonoma County?

ChatGPT’s answer: Napa County has an area of approximately 789 square kilometers (305 square miles), while Sonoma County is much larger, with an area of approximately 4,031 square kilometers (1,555 square miles). So, Sonoma County is more than five times larger than Napa County.

According to the U.S. Census Bureau, Napa County is 789 square miles, and Sonoma County is 1,768 square miles. So in the case of Napa, the chatbot mistook kilometers for miles, and in the case of Sonoma, it was just flat wrong. Its answer sounded good. And if I hadn’t double-checked, I might have believed it. Honestly, living here, Sonoma County feels five times as big when considering driving times to traverse. But that’s more a function of topography than geography.

My final question was, “What are the biggest wineries in Sonoma County?”

On this one, ChatGPT offered a more hedged answer, as it usually does if you ask it anything with even the slightest smidgen of controversy.

ChatGPT’s answer: Sonoma County is home to many wineries, both large and small, and it's difficult to determine the exact size of each one. However, here are some of the largest wineries in Sonoma County based on their annual production volume: Gallo Family Vineyards, Kendall-Jackson Winery, Francis Ford Coppola Winery, Rodney Strong Vineyards, Cline Cellars, Korbel Champagne Cellars, Beringer Vineyards, Simi Winery, E & J Gallo Winery, Jackson Family Wines.

But according to Wine Business Monthly, the biggest Sonoma County wineries are Jackson Family Wines, E&J Gallo, TIAA/Silverado Investment Management Group, Treasury Wine Estates, Rodney Strong Vineyards, Ferrari-Carano Vineyards, Constellation Brands, Sonoma-Cutrer Vineyards, Foley Family Wines, Sangiacomo Family Vineyards.

ChatGPT listed just three of the 10 biggest wineries in Sonoma County. It counted Jackson and E&J Gallo twice and included several wineries that shouldn’t be on the list.

Most of the time, the chatbot can’t determine reliable sources. It’s good at grammar, spelling, usage and syntax. Its answers look great at first glance, particularly if it’s a topic you’re not familiar with. But it’s pretty much just making bad information sound good. Basically, it’s just a really good liar.

If we rely on ChatGPT for answers, we think we’re getting the truth and may make poor decisions based on misinformation. Which seems an easy problem to manage. Just direct your questions to a sommelier instead of ChatGPT. But that assumes there are sommeliers to ask. ChatGPT is free. And therein lies the rub. Misinformation is cheaper and just as profitable as factual information.

Even the downside of misinformation costs less. While Fox News agreed to pay Dominion Voting Systems $787.5 million to settle a defamation claim, the cable network is unlikely to change its ways anytime soon, because its polemic pulpit ultimately generates more cash than it costs to settle the damages. The net-net is fake news is positive.

Which brings us to my fear of this AI-charged brave new world. If it’s cheaper to generate grammatically correct fluff, what’s the incentive for getting the facts straight? ChatGPT puts the ability to unknowingly generate misinformation in everyone’s hands. You don’t even need to trade in conspiracies or controversies. All you need is a profit motive.

If accuracy is just too expensive, what’s to stop the world from using ChatGPT to recommend the best food pairings with wines from a Sonoma County AVA that doesn’t exist?

Vintner Ross Halleck makes award-winning pinot noir at Halleck Vineyard in Sebastopol.

You can send letters to the editor to letters@pressdemocrat.com.