Please include local AI
k
kajuberdut
As a 15 year veteran of the software industry I'm not really interested in building a workflow that involves an online service that will undergo enshittification or just outright fail in a year or two.
I'm running exllamav2 and llama.cpp and I'm doing a lot of cool things with them. I would love to see some content around these tools especially now that pretty much anybody with a gaming GPU can run Gemma 2 9B or anybody with a modern smart phone can run Phi 3.5 Mini Instruct. These are the real future of AI.
L
Louis
Interesting. The only problem I see is the variation in response with the same prompt depending on the models, which are much more “varied” with open source AI (by that I mean there's so many open source models vs closed ones out there). Which makes most prompt useless in this case (or at least you'll need to iterate/make variations of them).
Will see if we can do a few tutorials on open source models or at least let people decide which model they want to use (which is possible with almost every tutorial we made, you'd just have to modify the prompt)
k
kajuberdut
Louis Thanks. It's fair that the local models each have their own prompt template, system prompt capabilities, and are each somewhat unique in their output. Also, agree this does add a layer of complexity but it is also a massive advantage since those models are able to be swayed towards a particular style or format in ways an "out of the box" system prompt will not allow.
Perhaps, including the prompt format and nuances in the tutorial is re-inventing the wheel?
In the interest of not making the tutorial writer lose their mind, you could take the approach of writing the steps at a more abstract/high-level and then linking the existing documentation on how to craft prompts for open source local models.
For example:
Command R / R+: https://docs.cohere.com/docs/crafting-effective-prompts
Lamma 3.1: https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1#prompt-format
L
Louis
kajuberdut: I'll need more time to think about it to be honest. In my opinion most people just want a prompt to copy & paste and if that doesn't work then they will probably start thinking about modifying the prompt.
I will think about a new "prompting" system that would be more abstract/future proof, but because most people are looking for simplicity and 0.1% will actually use an open source model, I can't guarantee it's going to be my absolute priority right now.