Man, I remember the first time I heard you explaining the concept of MashMaker 15 years ago at Intel, I was windblown, just like with this post. The clarity an succinctness in the way you present ideas and concepts is unparalleled. Besides your predictions, this is the best explanation on how Google works I've ever read.
This is a bit of an aside, but one likely outcome seems to be a nosedive in AdSense rev—both to Google and bloggers / content farms?
If you take a category like cooking or travel, being able to ping a bot with questions seems like a *much* better user experience. It can strip out just the recipe I need, and then I can ask it contextual questions about ingredients or prep methods in a messy / custom progression etc.
I wonder how this changes the economics on both sides of the table? Eg. travel bloggers will presumably see a lot fewer clicks on new content, and will thus take a dive on AdSense and referral links etc. Google would also lose rev, along with fresher content to incorporate.
The main model of content monetization has long been “win a click through search, then convert some % to subs, purchases, etc”. But if people never land on your page to begin with coz they’re just querying the content in aggregate form...
Yes. That's a real issue, and is something that was much discussed when we launched featured snippets - would the shift to direct answers reduce ad revenue to linked sites so much that we would damage the health of the sites Google depends on to answer user questions.
In the case of featured snippets we did an analysis that showed that (at least for question answering), we were actually less dependent on ad-funded sites than one might think. The best answers were usually on wikipedia, non-profit sites, gov sites, brand-building sites, or sites written by someone to increase their prestige. Content on purely ad-funded sites is generally lower quality and less suitable for being cited as a reputable source.
I also interviewed authors of high quality books and found that, in general, the authors of the best non-fiction books write mostly for prestige - either academic prestige, future consulting gigs, social access, or just personal satisfaction.
ChatGPT definitely changes things, but I expect some of the same principles to hold. In general, the best non-fiction sources write for prestige rather than ad revenue, and will likely continue to do so provided ChatGPT cites them, and they have some way of knowing they are gaining prestige by being cited.
I haven't tried this yet, but instinctively it would feel most natural to me to use a chatbot in cases where the prior alternatives were "talk to a human agent" or "navigate this highly structured vertical search interface." So travel queries, restaurant reservations, etc. Google has spent a ton of effort getting their structured vertical interfaces right, and I wonder if chatbots will outcompete them in those verticals or if Bard will actually complement them nicely (because at the end of the task what you want to do is take a structured, parameterized action based on the outcome of the chat).
My sense is that it's still too early to tell what chatbots will be most useful for. I keep seeing people sharing examples of Twitter of cool things you can use ChatGPT for that I wouldn't have thought of. That's why I love Poe - it lets me talk to a chatbot and shows me examples of cool use cases that other people have found.
I think you are right that a lot of use cases for chatbots will be cases where you would have previously talked to a human. E.g. financial advisor, junior lawyer, copy-editor, teacher.
My guess is that the super-structured stuff is currently done well enough by Google that there isn't /that/ much opportunity for chatbots to do a better job, but I wouldn't be surprised if there is some big win there that I'm missing.
I think part of the challenge with LLMs is that they learn to "think" or "predict" the spoken word across the entire distribution of humanity. The training data will include a lot of racist, paranoid and wrong content. So it's capable of expressing all though different types of thinking. The initial prompt tries to filter it to an acceptable type of thinking but it's not capable of doing that reliably.
I agree with what you said about it weighting recent conversation too much. I think that once the LLM has gone down the rabbit hole, it's not able to get out again. Perhaps we should have a second LLM with constantly reset history, that's assessing the first LLM. Similar to the human safety factors of dual pilots. One performs the actions while the other assesses them. I don't know if this is the right answer but I'm sure there will be lots of experimentation of it in the years to come.
Another big question is how aware the chatbot would be of the rest of your life? Your intents, goals, calendar and communications. With ChatGPT, you're having these isolated conversations without permanence, indeed the longer the conversations the more likely that Bing Chat shows strange behaviour: https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html. Think a moment of a human assistant and the things they can do so much better. That requires so much in terms of privacy, trust and long lived interactions that it will take a while to emerge.
Rob, I'd add that was an excellent summary of Google and how it operates. They've been quite assiduous in making the most of the opportunity.
Just as Bing Chat has access to search, I can imagine that future assistant-ish assistants would have access to your personal data (email, calendar, docs, etc) and the ability to update their own internal database of information about your intents/goals etc based on your conversation.
I suspect that part of the reason why Bing Chat gets weird on longer chats is that it weights too much on the recent conversation rather than its intended assistant personality when generating text completions. Kind of as if you decided what to write based entirely on the last paragraph you had seen yourself write, with no higher sense of long-term self to guide you. I'd expect that to improve.
Man, I remember the first time I heard you explaining the concept of MashMaker 15 years ago at Intel, I was windblown, just like with this post. The clarity an succinctness in the way you present ideas and concepts is unparalleled. Besides your predictions, this is the best explanation on how Google works I've ever read.
Thanks. I appreciate it :-)
This is a bit of an aside, but one likely outcome seems to be a nosedive in AdSense rev—both to Google and bloggers / content farms?
If you take a category like cooking or travel, being able to ping a bot with questions seems like a *much* better user experience. It can strip out just the recipe I need, and then I can ask it contextual questions about ingredients or prep methods in a messy / custom progression etc.
I wonder how this changes the economics on both sides of the table? Eg. travel bloggers will presumably see a lot fewer clicks on new content, and will thus take a dive on AdSense and referral links etc. Google would also lose rev, along with fresher content to incorporate.
The main model of content monetization has long been “win a click through search, then convert some % to subs, purchases, etc”. But if people never land on your page to begin with coz they’re just querying the content in aggregate form...
Yes. That's a real issue, and is something that was much discussed when we launched featured snippets - would the shift to direct answers reduce ad revenue to linked sites so much that we would damage the health of the sites Google depends on to answer user questions.
In the case of featured snippets we did an analysis that showed that (at least for question answering), we were actually less dependent on ad-funded sites than one might think. The best answers were usually on wikipedia, non-profit sites, gov sites, brand-building sites, or sites written by someone to increase their prestige. Content on purely ad-funded sites is generally lower quality and less suitable for being cited as a reputable source.
I also interviewed authors of high quality books and found that, in general, the authors of the best non-fiction books write mostly for prestige - either academic prestige, future consulting gigs, social access, or just personal satisfaction.
ChatGPT definitely changes things, but I expect some of the same principles to hold. In general, the best non-fiction sources write for prestige rather than ad revenue, and will likely continue to do so provided ChatGPT cites them, and they have some way of knowing they are gaining prestige by being cited.
I haven't tried this yet, but instinctively it would feel most natural to me to use a chatbot in cases where the prior alternatives were "talk to a human agent" or "navigate this highly structured vertical search interface." So travel queries, restaurant reservations, etc. Google has spent a ton of effort getting their structured vertical interfaces right, and I wonder if chatbots will outcompete them in those verticals or if Bard will actually complement them nicely (because at the end of the task what you want to do is take a structured, parameterized action based on the outcome of the chat).
My sense is that it's still too early to tell what chatbots will be most useful for. I keep seeing people sharing examples of Twitter of cool things you can use ChatGPT for that I wouldn't have thought of. That's why I love Poe - it lets me talk to a chatbot and shows me examples of cool use cases that other people have found.
I think you are right that a lot of use cases for chatbots will be cases where you would have previously talked to a human. E.g. financial advisor, junior lawyer, copy-editor, teacher.
My guess is that the super-structured stuff is currently done well enough by Google that there isn't /that/ much opportunity for chatbots to do a better job, but I wouldn't be surprised if there is some big win there that I'm missing.
I think part of the challenge with LLMs is that they learn to "think" or "predict" the spoken word across the entire distribution of humanity. The training data will include a lot of racist, paranoid and wrong content. So it's capable of expressing all though different types of thinking. The initial prompt tries to filter it to an acceptable type of thinking but it's not capable of doing that reliably.
I agree with what you said about it weighting recent conversation too much. I think that once the LLM has gone down the rabbit hole, it's not able to get out again. Perhaps we should have a second LLM with constantly reset history, that's assessing the first LLM. Similar to the human safety factors of dual pilots. One performs the actions while the other assesses them. I don't know if this is the right answer but I'm sure there will be lots of experimentation of it in the years to come.
Another big question is how aware the chatbot would be of the rest of your life? Your intents, goals, calendar and communications. With ChatGPT, you're having these isolated conversations without permanence, indeed the longer the conversations the more likely that Bing Chat shows strange behaviour: https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html. Think a moment of a human assistant and the things they can do so much better. That requires so much in terms of privacy, trust and long lived interactions that it will take a while to emerge.
Rob, I'd add that was an excellent summary of Google and how it operates. They've been quite assiduous in making the most of the opportunity.
Good question.
Just as Bing Chat has access to search, I can imagine that future assistant-ish assistants would have access to your personal data (email, calendar, docs, etc) and the ability to update their own internal database of information about your intents/goals etc based on your conversation.
I suspect that part of the reason why Bing Chat gets weird on longer chats is that it weights too much on the recent conversation rather than its intended assistant personality when generating text completions. Kind of as if you decided what to write based entirely on the last paragraph you had seen yourself write, with no higher sense of long-term self to guide you. I'd expect that to improve.
I'm pleased you appreciated the post.