Rendered at 12:10:49 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
N_Lens 6 hours ago [-]
I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable.
I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet).
It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.
chris_money202 2 hours ago [-]
I think future is probably more similar to speculative execution (inference/decoding). A small LLM is used to speculate and a large LLM is used to confirm if needed. If the small LLM is accurate enough on N tokens it’s cheap for the large LLM to say looks good and keep moving along.
hiyfsch 1 hours ago [-]
[dead]
visarga 3 hours ago [-]
I think the harness and local context should supply that missing piece between general model and bespoke application. Each application has its own context and action quirks that don't generalize well. Maybe it's just 5% but that is genuinely specific. So its rightful place is in context engineering.
> It's actually how organic brains work - specialized tasks are offloaded to local cortical columns.
How are small isolated language models more similar to that than MoE in LLMs?
simianwords 4 hours ago [-]
Right MoE is a tradeoff between efficiency and intelligence.
andy99 4 hours ago [-]
General purpose models are always more robust and generally better than smaller narrower models. My bet is that compute will catch up and any “small” model will still be generally capable, just smaller than sota, rather than intentionally narrow. The exception would be for very well defined tasks where the data distribution never varies, but these are rare and don’t really need “AI” anyway when they do exist.
swiftcoder 4 hours ago [-]
> General purpose models are always more robust and generally better than smaller narrower models
I feel like this is just the marketing conflation of AI=LLM, versus regular old ML? We're never going to need to deploy a full reasoning model on a low-power device just to do some fancy image recognition in the field. Specialised ML models are just intrinsically able to be a lot more efficient than their generalist equivalents
plastic-enjoyer 4 hours ago [-]
> General purpose models are always more robust and generally better than smaller narrower models.
What do you mean with more robust?
3 hours ago [-]
looofooo0 5 hours ago [-]
What about recent models providing correct proofs to open math problems?
TJSomething 5 hours ago [-]
I haven't tried it, but I saw Leanstral, an LLM specialized in writing Lean proofs, posted on HN recently and it claims to outperform some larger general purpose models. It didn't beat Claude Opus, but it seems to do decently at one tenth the cost. It's plausible that further research could yield other models that are smaller and more effective at limited tasks, reversing the trend of ever growing models.
16bitvoid 5 hours ago [-]
What about it?
simianwords 4 hours ago [-]
No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds.
Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classifiers?
dofm 43 minutes ago [-]
> No this will never work.
This bet is too early.
> Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier to scrape everything rather than spend enormous effort (with tools that did not really exist) to partition the training set. Any number of other "because"s.
It's just what they are doing now and it showed the earliest results.
LLMs are still less intelligent than rats, which have tiny brains.
fasterik 4 hours ago [-]
Your examples (math, literature) involve natural language. It stands to reason that a general language model will be more competitive in those domains. If you want examples of successful domain-specific models, look at AlphaZero and AlphaFold. LLMs aren't anywhere close to achieving that level of competence at abstract strategy games or protein folding.
"This will never work" is a pretty confident assertion for a field that's so young and rapidly evolving.
simianwords 4 hours ago [-]
> I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet).
This is what the parent said. AGI won't rise out of AlphaZero and AlphaFold in the same way AGI won't rise out of Houdini chess engine. This is the industry consensus.
dofm 42 minutes ago [-]
AGI is a macguffin (or a shaggy dog) for a story told to investors. It has never been plausible on the timescales suggested and it almost certainly will not emerge from LLMs.
fasterik 3 hours ago [-]
>AGI won't rise out of AlphaZero and AlphaFold in the same way AGI won't rise out of Houdini chess engine.
That's a straw man. Nobody thinks AGI will rise out of domain-specific systems. The question is whether domain-specific systems are necessary for AGI.
Of course, the problem is that AGI isn't a well-defined concept. But if we define it as achieving superhuman performance across several hundred domains where there are objective measures of success, it doesn't seem far-fetched to predict that it will involve some general reasoning system paired with a bunch of specialized modules.
simianwords 3 hours ago [-]
The parent said
> I don't foresee AGI arising out training bigger LLMs
I agree that AGI will involve tool usage but not only involving domain specific AI models.
But lets try to find the discriminating point in the discussion - do you believe AGI will necessarily involve training bigger LLM's or not?
I believe they are necessary. WBU?
fasterik 3 hours ago [-]
You're still intentionally misreading the OP's statement. If you read it again in context, they're clearly saying that they think training bigger LLMs is not sufficient. I think I agree with that statement, but my confidence is pretty low.
No, I don't think LLMs are necessary for AGI at all. I think there are multiple paths to AGI, some of which involve LLMs and some which don't.
thereitgoes456 4 hours ago [-]
DeepMind did release a math specific model. And OpenAI has released a coding specific model.
The answer to your question is “because the market isn’t big enough”, not because it doesn’t work. Why would knowing about 2019 internet memes help you in any way at coding?
simianwords 4 hours ago [-]
> And OpenAI has released a coding specific model
They did and retracted it because they found that GPT 5.5 beat codex pareto optimally. This keeps happening.
> because the market isn’t big enough
Huuh? market isn't big enough for AGI? The parent suggested that AGI would emerge from this process.
piotrekno1 1 hours ago [-]
[flagged]
tim-fan 10 hours ago [-]
Is anyone making LLM-in-a-box for emergency supply kits yet?
I feel that would be handy in all sorts of situations when networks are down.
Terr_ 9 hours ago [-]
> LLM-in-a-box for emergency
For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.)
In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful?
Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?
skybrian 9 hours ago [-]
You will probably want a search engine though. Perhaps a small LLM would work well as a component for that?
visarga 3 hours ago [-]
> You will probably want a search engine though.
The search engine is indeed the last missing component from a sovereign stack. But I think this could be solved locally with little cost. Instead of indexing content on the web we should be indexing sources themselves - where to look for X? - like forums, blogs, docs, feeds, and specialized search engines. We could collectively amass millions of these search stubs that can be used by local models to go and fetch fresh information from the source directly. This means separating the routing layer from the information layer, we don't need to keep information cached from the whole internet locally. The search stubs could fit in a few GB about same size with the local LLM. The cool thing is that sources change much slower than information itself, so the search stub database could be refreshed at a slower pace. We could combine a few million generic stubs with a few hundred personal stubs generated from our own activities. It is trivial to generate these stubs by piggy backing on frontier models.
iamflimflam1 4 hours ago [-]
You may benefit from an embedding approach for semantic search. Not sure what an LLM would give you on top of that.
crsx_ 3 hours ago [-]
If the emergency is in a foreign country, being able to communicate with locals would likely be a benefit - and a domain specific trained model could translate better than general purpose translators.
In general, I think speech as input/output is under-explored. In the emergency scenario, in a stressful environment, having an expert in your ear you can talk to should work much better than having a big manual book to look up specific cases.
Lio 3 hours ago [-]
I think an LLM would give a "conversational" experience to search.
That's handy for situations where you might not really understand what you need to search for. Any search system that can ask you clarifying questions is going to be a big improvement.
Or where you need to combine several steps together but you don't yet know what those steps are.
There's probably other technologies that could do that, requiring lower resources but they'll come with different trade-offs around configuration.
Just having a Raspberry PI, a offline copy wikipedia and a RAG enabled small LLM would be quite useful or at least entertaining if you have to go off grid.
rtpg 6 hours ago [-]
grep works well!
zmgsabst 7 hours ago [-]
You can run a small model off a home generator — so in an emergency, you’d turn on both the generator and information service, eg, a mesh for “quick” responses querying that huge collection of information.
That way your machine that, eg, normally plays video games or does AI work can support relief efforts by supporting emergency response IT. You don’t need to mothball the machine, just have an “emergency” boot USB than can run the services from your home generator.
You don’t even need to bring it with you: turn it on and leave it “best effort” at home, while you continue to use it via WAN.
Terr_ 5 hours ago [-]
I feel this is going into increasingly-unlikely mixes of constraints and needs in order to try to keep a "wouldn't it be cool if" hypothetical-tool dream alive. [0]
But OK, let's assume that: The power is out, but you have a generator with so much fuel you can run a desktop just fine; Your neighborhood will somehow make a mesh network; Your neighbors need some already stored information and the best solution for that is texting a chatbot rather than a survival/emergency handbook or Wikipedia; Your mesh-network will also be good enough to match the time-sensitivity of the questions.
Under those assumption, which of these sounds better?
1. Buying an "LLM-in-a-box for emergency supply kits", which you deploy so that your neighbors can ask questions (text over the mesh) of the offline chatbot.
2. Buying a satellite internet transciever for your emergency supply kit, so that your neighbors can ask questions of a much better chatbot and communicate with human experts, their worried relatives, and coordinate with rescue/relief efforts...
I’m only out the cost of the drive, which is like $40 and doesn’t require anybody on the other side cooperate with me.
- - -
More broadly…
You call it unlikely mixes, but we see it all the time:
- people already have a computer for gaming or work
- people (ie, “preppers” like we’re discussing) buy a generator for emergencies
- local emergency response sets up mesh networking during disasters, both official and unofficial
Have you ever tried to use a handbook you’re not intimately familiar with during an emergency? It’s rough.
For personal preparedness, nothing replaces familiarity and practice — eg, weekend survival trips and reading your manual ahead of time.
But for providing information in a random lookup manner to unpracticed people who weren’t prepared? Yes, I think an LLM/chatbot is the practical way to operationalize all that information which you stored (eg, survival guides or machine manuals).
Also, it’s unlikely a general purpose chatbot would be superior at survival advice to one specialized for that purpose — and indeed, is likely to refuse your questions as “unsafe” or “criminal”.
swiftcoder 4 hours ago [-]
> I’m only out the cost of the drive, which is like $40 and doesn’t require anybody on the other side cooperate with me.
At current prices you are also out about $4k for a Spark to actually run the inference on, if you want a full LLM in a low-power package.
In general, I'm not sure why one would want to pin your survival to an expensive, hallucination-prone data source, when an offline copy of wikipedia with a little vector search attached to a Raspberry Pi can fulfil the same role...
weikju 8 hours ago [-]
> If you do carry it to an enclave of civilization that has the right power, are they going to let you run it when it cuts into light for surgeons or heat to sterilize water?
Knowing humans? They'd probably take it by force and run it for themselves instead of providing light and heat to surgeons and water sterilizers...
/daily dose of cynism
SwellJoe 9 hours ago [-]
This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/
But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.
dofm 27 minutes ago [-]
The 12B QAT model is really overlooked because the tech industry has been so desperate for the LLM bet to play out to "product market fit" (which means please IPO now) that it has become convinced that coding models are the only things that matter.
bluerooibos 46 minutes ago [-]
> Is anyone making LLM-in-a-box for emergency supply kits yet?
Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful.
"So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"
I've been mulling over a good use of a large philanthropy spend in the next decade, and I would love to build a bunch of hardware "oracles" that include an LLM. Ideally solid state, visual/audio, solar + usb-c, so, good in a lot of doomsday scenarios as well as just out hiking. It's a fun thought experiment. I imagine making like 1 million of them, they could be sold and genuinely useful, but also given away; once owned, you could use them, or store and put in an emergency box, bury next to the 10k year clock.. a lot of possibilities.
adrianN 7 hours ago [-]
I feel like you could get a lot more quality of life improvement for more people with the money if you spent it on low tech solutions, eg more efficient cooking stoves for people still cooking with biomass, or solar microgrids for areas without electricity.
vessenes 2 hours ago [-]
No doubt there are a hundred more direct things one could do. On the other hand this would include instructions for all those things and could advise on the building! I think it would be a nice thing to have in the world.
sumitkumar 5 hours ago [-]
electricity outage and battery running out is the end game for any real prolonged external emergency. Internet connection is just the soft edge.
rasz 6 hours ago [-]
Smallest local model able to work with offline wikipedia dump would be one step above just having an offline wikipedia dump.
cdnsteve 10 hours ago [-]
Can you expand what you mean?
wahnfrieden 9 hours ago [-]
They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.
dofm 19 minutes ago [-]
> They want to ask the iOS Foundation model (frontier on device intelligence for something small)
This is a bit of a straw man, TBH.
For one thing, "LLM-in-a-box" doesn't necesssarily imply a device as small as a phone.
For another, you'd need to convince people that the iOS Foundation model is the "frontier" of LLMs that run on phones when it is really not. AFAIK it is noticeably outperformed by the Gemma 4 E2B model and certainly the E4B.
Whether this idea (LLMs for emergency/survival scenarios) has value, I don't know, so I am not offering an opinion, but you should approach it with a good faith argument.
I am an LLM cynic but I suppose if I was to be without connectivity but with power for a while, a device with the Gemma 4 E2B or E4B model on it might be helpful or interesting to have. If such a device had the 12B QAT model on it, that really would cross the line to utility. Not sure it has value in the OP's scenario, still.
burgerone 6 hours ago [-]
This is HN, not Reddit.
egormakarov 3 hours ago [-]
Can't wait to be killed by my toaster because some sexy mossad agent seduced it.
a96 3 hours ago [-]
Have you ever tried to indulge an all-consuming urge to kill when you don't have opposable thumbs? Or hands? Or anything other than a bread slot?
masfuerte 2 hours ago [-]
Fire!
sscaryterry 1 hours ago [-]
This made me laugh out loud :)
monkeydust 5 hours ago [-]
Where is a good place to start with training SLM these days if you don't have the compute locally?
calgoo 2 hours ago [-]
I rent cheap GPU based instances by the hour and run on those. Nothing fancy, but $20 can get you a decent amount of compute on a A6000 or H100.
bix6 8 hours ago [-]
Has anyone used the Rx Scanner mentioned in the opening?
99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.
jdonaldson 9 hours ago [-]
I think neuro-symbolic AI has a lot of potential here, since small models can handle a lot of conversational inputs, while relying on wired-in solvers for more complex symbolic math/computation needs. https://jjd.io/posts/swollm-bbh-leaderboard.html
enoint 10 hours ago [-]
Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.
fpauser 5 hours ago [-]
SLMs for the rescue!
mountainriver 7 hours ago [-]
I looked into this a bit but unfortunately because of starlink most of this won’t be needed
I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet).
It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.
I have a long-ass post about how this could be implemented. https://old.reddit.com/r/VisargaPersonal/comments/1um9uyv/st...
How are small isolated language models more similar to that than MoE in LLMs?
I feel like this is just the marketing conflation of AI=LLM, versus regular old ML? We're never going to need to deploy a full reasoning model on a low-power device just to do some fancy image recognition in the field. Specialised ML models are just intrinsically able to be a lot more efficient than their generalist equivalents
What do you mean with more robust?
Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classifiers?
This bet is too early.
> Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier to scrape everything rather than spend enormous effort (with tools that did not really exist) to partition the training set. Any number of other "because"s.
It's just what they are doing now and it showed the earliest results.
LLMs are still less intelligent than rats, which have tiny brains.
"This will never work" is a pretty confident assertion for a field that's so young and rapidly evolving.
This is what the parent said. AGI won't rise out of AlphaZero and AlphaFold in the same way AGI won't rise out of Houdini chess engine. This is the industry consensus.
That's a straw man. Nobody thinks AGI will rise out of domain-specific systems. The question is whether domain-specific systems are necessary for AGI.
Of course, the problem is that AGI isn't a well-defined concept. But if we define it as achieving superhuman performance across several hundred domains where there are objective measures of success, it doesn't seem far-fetched to predict that it will involve some general reasoning system paired with a bunch of specialized modules.
> I don't foresee AGI arising out training bigger LLMs
I agree that AGI will involve tool usage but not only involving domain specific AI models.
But lets try to find the discriminating point in the discussion - do you believe AGI will necessarily involve training bigger LLM's or not?
I believe they are necessary. WBU?
No, I don't think LLMs are necessary for AGI at all. I think there are multiple paths to AGI, some of which involve LLMs and some which don't.
The answer to your question is “because the market isn’t big enough”, not because it doesn’t work. Why would knowing about 2019 internet memes help you in any way at coding?
They did and retracted it because they found that GPT 5.5 beat codex pareto optimally. This keeps happening.
> because the market isn’t big enough
Huuh? market isn't big enough for AGI? The parent suggested that AGI would emerge from this process.
I feel that would be handy in all sorts of situations when networks are down.
For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.)
In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful?
Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?
The search engine is indeed the last missing component from a sovereign stack. But I think this could be solved locally with little cost. Instead of indexing content on the web we should be indexing sources themselves - where to look for X? - like forums, blogs, docs, feeds, and specialized search engines. We could collectively amass millions of these search stubs that can be used by local models to go and fetch fresh information from the source directly. This means separating the routing layer from the information layer, we don't need to keep information cached from the whole internet locally. The search stubs could fit in a few GB about same size with the local LLM. The cool thing is that sources change much slower than information itself, so the search stub database could be refreshed at a slower pace. We could combine a few million generic stubs with a few hundred personal stubs generated from our own activities. It is trivial to generate these stubs by piggy backing on frontier models.
In general, I think speech as input/output is under-explored. In the emergency scenario, in a stressful environment, having an expert in your ear you can talk to should work much better than having a big manual book to look up specific cases.
That's handy for situations where you might not really understand what you need to search for. Any search system that can ask you clarifying questions is going to be a big improvement.
Or where you need to combine several steps together but you don't yet know what those steps are.
There's probably other technologies that could do that, requiring lower resources but they'll come with different trade-offs around configuration.
Just having a Raspberry PI, a offline copy wikipedia and a RAG enabled small LLM would be quite useful or at least entertaining if you have to go off grid.
That way your machine that, eg, normally plays video games or does AI work can support relief efforts by supporting emergency response IT. You don’t need to mothball the machine, just have an “emergency” boot USB than can run the services from your home generator.
You don’t even need to bring it with you: turn it on and leave it “best effort” at home, while you continue to use it via WAN.
But OK, let's assume that: The power is out, but you have a generator with so much fuel you can run a desktop just fine; Your neighborhood will somehow make a mesh network; Your neighbors need some already stored information and the best solution for that is texting a chatbot rather than a survival/emergency handbook or Wikipedia; Your mesh-network will also be good enough to match the time-sensitivity of the questions.
Under those assumption, which of these sounds better?
1. Buying an "LLM-in-a-box for emergency supply kits", which you deploy so that your neighbors can ask questions (text over the mesh) of the offline chatbot.
2. Buying a satellite internet transciever for your emergency supply kit, so that your neighbors can ask questions of a much better chatbot and communicate with human experts, their worried relatives, and coordinate with rescue/relief efforts...
[0] https://xkcd.com/2128/
I’m only out the cost of the drive, which is like $40 and doesn’t require anybody on the other side cooperate with me.
- - -
More broadly…
You call it unlikely mixes, but we see it all the time:
- people already have a computer for gaming or work
- people (ie, “preppers” like we’re discussing) buy a generator for emergencies
- local emergency response sets up mesh networking during disasters, both official and unofficial
Have you ever tried to use a handbook you’re not intimately familiar with during an emergency? It’s rough.
For personal preparedness, nothing replaces familiarity and practice — eg, weekend survival trips and reading your manual ahead of time.
But for providing information in a random lookup manner to unpracticed people who weren’t prepared? Yes, I think an LLM/chatbot is the practical way to operationalize all that information which you stored (eg, survival guides or machine manuals).
Also, it’s unlikely a general purpose chatbot would be superior at survival advice to one specialized for that purpose — and indeed, is likely to refuse your questions as “unsafe” or “criminal”.
At current prices you are also out about $4k for a Spark to actually run the inference on, if you want a full LLM in a low-power package.
In general, I'm not sure why one would want to pin your survival to an expensive, hallucination-prone data source, when an offline copy of wikipedia with a little vector search attached to a Raspberry Pi can fulfil the same role...
Knowing humans? They'd probably take it by force and run it for themselves instead of providing light and heat to surgeons and water sterilizers...
/daily dose of cynism
But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.
Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful.
"So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"
Put that on a spare phone
This is a bit of a straw man, TBH.
For one thing, "LLM-in-a-box" doesn't necesssarily imply a device as small as a phone.
For another, you'd need to convince people that the iOS Foundation model is the "frontier" of LLMs that run on phones when it is really not. AFAIK it is noticeably outperformed by the Gemma 4 E2B model and certainly the E4B.
https://blog.google/innovation-and-ai/technology/developers-...
Here is a common-or-garden youtube video that includes a demonstration of how much better the E2B is:
https://www.youtube.com/watch?v=sTxyBUbdZcA
Whether this idea (LLMs for emergency/survival scenarios) has value, I don't know, so I am not offering an opinion, but you should approach it with a good faith argument.
I am an LLM cynic but I suppose if I was to be without connectivity but with power for a while, a device with the Gemma 4 E2B or E4B model on it might be helpful or interesting to have. If such a device had the 12B QAT model on it, that really would cross the line to utility. Not sure it has value in the OP's scenario, still.
https://rxall.net/rxscanner/
I've been working on small local models for years with txtai (https://github.com/neuml/txtai). I've published close to 100 models that can run local for RAG, Agents, Vector Search and more (https://huggingface.co/NeuML/collections).