After training LLMs on the whole world, prompting them for zero-shot, finetuning them on specialized domains, what is the next stage?
The goal is clear: adapting models further and further.
I can see two competitors there: website finetuning and recommenders.
– Recommenders –
Recommenders are basically finetuned on users. They already use embeddings, are effective, and do not need labeling thanks to user events. But privacy is a problem.
– Website finetuning –
On the other hand, Website finetuning is performed on the site content, especially vocabulary. But labeling is a problem. We can use LLMs to generate labels (questions), but is it not a chiken/eggs loop?
