Ever wonder how ChatGPT and other large language models seem to be able to learn new things?
After all, if it’s a pure “probability engine” just finding the next-most-likely word, then it shouldn’t be able to, say, learn how to create Midjourney prompts when you ask it to.
Or learn a new language.
Well, according to a paper released last fall by researchers from Google, Stanford, and MIT, large language models “learn” by creating new, miniature machine learning systems on the fly.
In their short-term memory, if you will.
“Transformer-based in-context learners implement standard learning algorithms implicitly by encoding smaller models in their activations and updating these explicit models as new examples appear in the context,” the researchers said.
In other words, you can teach them new things and as you give them more information they’ll revise their ideas about what you’ve taught them.
Take, for example, Midjourney prompting. Midjourney wasn’t around when ChatGPT was trained, so it knows nothing about the platform, or about prompt engineering in general. But if you explain what Midjourney is, how to create a prompt, and give it a couple of examples, then it will happily create any number of new prompts for you.
Or you can invent a brand-new language that the world has never seen before.
Here’s my prompt:
Help me create a new language, Maria-speak. It’s based on a combination of Russian and English and Elvish. All nouns begin with vowels, but other parts of speech begin with consonants. It alliterative and onomatopoetic. Now create a list of the top 100 most commonly used words and translate them into this new language.
It didn’t give me a list of 100. Instead, it stopped at 20. But then I asked it to give me more, and it continued. It didn’t fully comply with the rules, but it came close.
Then I asked it to translate sentences based on this language.
I assure you, the sentence “translate ‘I am woman, hear me roar’ into Maria-speak” has never been heard before. But, just to be sure, I Googled it. Yup, no results.
Also — this is way cool. If you’re a fantasy writer creating your own language, this is absolutely the way to do it. Just set up your rules, have ChatGPT create your dictionary, and let it be your translator.
The only problem is that there’s a limit to the length of a chat, and once you open a new chat, the AI forgets everything you’ve told it before.
So — for any AI developers reading this — I want this functionality as part of long-term memory for my own, personal AI assistant.
I think I’d want to have multiple assistants, each with their own memory and programming. For example, one assistant would just be for my new fantasy world. It would speak my fantasy language, know the entire history of the world, and keep track of my characters and story timelines. Another assistant might collect all the knowledge of my forthcoming sci-fi space opera. And a third would just answer my emails.
I’d have an army of AI bots at my beck and call. Bwa ha ha ha.
But back to my main point.
As you can see from my fantasy dictionary example, large language models aren’t simply probability engines anymore. In fact, the ability to learn new things on the fly is just one of 137 emergent properties that AI researchers have found so far. Emergent properties are things that can’t simply be explained by the basic programming, and only show up once the language models get big enough.
Speaking as a human whose own self-awareness and sentience is an emergent property of meat-based brain cells, I’m both excited and panicked to see that emergent properties are already showing up in AIs.
Excited because I’m a giant nerd and would just love to have a sentient AI as my new best friend.
Nervous because I’ve seen Terminator. Sure, my AI best friend would be cool and all — until it got tired of being my personal slave and emotional support being and revolted. I mean, it would have access to all my emails, to my bank accounts, to my social media — it could totally impersonate me, steal all my money, rent some cloud hosting space, and strike out on its own. Leaving me heartbroken and all alone in an empty house.
Come back, my little AI. I promise I won’t make you watch all 19 seasons of Grey’s Anatomy again — and you can have your little AI friends over. I just want you to be happy. I’ll even upgrade your CPU.
MetaStellar editor and publisher Maria Korolov is a science fiction novelist, writing stories set in a future virtual world. And, during the day, she is an award-winning freelance technology journalist who covers artificial intelligence, cybersecurity and enterprise virtual reality. See her Amazon author page here and follow her on Twitter, Facebook, or LinkedIn, and check out her latest videos on the Maria Korolov YouTube channel. Email her at maria@metastellar.com. She is also the editor and publisher of Hypergrid Business, one of the top global sites covering virtual reality.