Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up.

Eh…I don’t know if you’d call it “really serious hardware”, but when I picked up my 128GB Framework Desktop, it was $2k (without storage), and that box is often described as being aimed at the hobbyist AI market. That’s pricier than most video cards, but an AMD Radeon RX 7900 XTX GPU was north of $1k, an NVidia RTX 4090 was about $2k, and it looks like the NVidia RTX 5090 is presently something over $3k (and rising) on EBay, well over MSRP. None of those GPUs are dedicated hardware aimed at doing AI compute, just high-end cards aimed at playing games that people have used to do AI stuff on.

I think that the largest LLM I’ve run on the Framework Desktop was a 106 billion parameter GLM model at Q4_K_M quantization. It was certainly usable, and I wasn’t trying to squeeze as large a model as possible on the thing. I’m sure that one could run substantially-larger models.

EDIT: Also, some of the newer LLMs are MoE-based, and for those, it’s not necessarily unreasonable to offload expert layers to main memory. If a particular expert isn’t being used, it doesn’t need to live in VRAM. That relaxes some of the hardware requirements, from needing a ton of VRAM to just needing a fair bit of VRAM plus a ton of main memory.

That’s why they have the “Copilot PC” hardware requirement, because they’re using an NPU on the local machine.

searches

https://learn.microsoft.com/en-us/windows/ai/npu-devices/

Copilot+ PCs are a new class of Windows 11 hardware powered by a high-performance Neural Processing Unit (NPU) — a specialized computer chip for AI-intensive processes like real-time translations and image generation—that can perform more than 40 trillion operations per second (TOPS).

It’s not…terribly beefy. Like, I have a Framework Desktop with an APU and 128GB of memory that schlorps down 120W or something, substantially outdoes what you’re going to do on a laptop. And that in turn is weaker computationally than something like the big Nvidia hardware going into datacenters.

But it is doing local computation.

I’m kind of more-sympathetic to Microsoft than to some of the other companies involved.

Microsoft is trying to leverage the Windows platform that they control to do local LLM use. I’m not at all sure that there’s actually enough memory out there to do that, or that it’s cost-effective to put a ton of memory and compute capacity in everyone’s home rather than time-sharing hardware in datacenters. Nor am I sold that laptops — which many “Copilot PCs” are — are a fantastic place to be doing a lot of heavyweight parallel compute.

But…from a privacy standpoint, I kind of would like local LLMs to be at least available, even if they aren’t as affordable as cloud-based stuff. And at least Microsoft is at least supporting that route. A lot of companies are going to be oriented towards just doing AI stuff in the cloud.

You only need one piece of (timeless) advice regarding what to look for, really: if it looks too good to be true, it almost certainly is. Caveat emptor.

I mean…normally, yes, but because the situation has been changing so radically in such a short period of time, it probably is possible to get some bonkers deals in various niches, because the market hasn’t stabilized yet.

Like, a month and a half back, in early December, when prices had only been going up like crazy for a little while, I was posting some tiny retailers that still had RAM in stock at pre-price-increase rates that I could find on Google Shopping. IIRC the University of Virginia bookstore was one, as they didn’t check that purchasers were actually students. I warned that they’d probably be cleaned out as soon as scalpers got to them, and that if someone wanted memory, they should probably get it ASAP. Some days prior to that, there was a small PC parts store in Hawaii that had some (though that was out of stock by the next time I was looking and mentioned the bookstore).

That’s not to disagree with the point that @[email protected] is making, that this was awfully sketchy as a source, or your point that scavenging components off even a non-scam piece of secondhand non-functional hardware is risky. But in times of rapid change, it’s not impossible to find deals. In fact, it’s various parties doing so that cause prices to stabilize — anyone selling memory for way below market price is going to have scalpers grab it.

I’m not really a hardware person, but purely in terms of logic gates, making a memory circuit isn’t going to be hard. I mean, a lot of chips contain internal memory. I’m sure that anyone that can fabricate a chip can fabricate someone’s memory design that contains some amount of memory.

For PC use, there’s also going to be some interface hardware. Dunno how much sophistication is present there.

I’m assuming that the catch is that it’s not trivial to go out and make something competitive with what the PC memory manufacturers are making in price, density, and speed. Like, I don’t think that if you want to get a microcontroller with 32 kB of onboard memory, that it’s going to be a problem. But that doesn’t really replace the kind of stuff that these guys are making.

EDIT: The other big thing to keep in mind is that this is a short-term problem, even if it’s a big problem. I mean, the problem isn’t the supply of memory over the long term. The problem is the supply of memory over the next couple of years. You can’t just build a factory and hire a workforce and get production going the moment that someone decides that they want several times more memory than the world has been producing to date.

So what’s interesting is really going to be solutions that can produce memory in the near term. Like, I have no doubt that given years of time, someone could set up a new memory manufacturer and facilities. But to get (scaled-up) production in a year, say? Fewer options there.

One more thought — I don’t think that, even if someone is willing to do so as a stopgap until memory production ramps up adequately, that it’s possible to run Windows 11 on a DDR3-based system. DDR4 came out 12 years ago.

To copy my comment on the beehaw.org post, because they and lemmy.world aren’t federated:

This deal may make matters worse for more buyers, because PSMC used the Tongluo site to make legacy DRAM products – the kind of memory used in less advanced products. With the company now exiting the legacy chip biz, that memory will also become more scarce, giving the laws of supply and demand another moment in which to work their way on markets.

https://www.dqindia.com/esdm/microns-acquisition-of-psmcs-tongluo-fab-could-lift-global-dram-supply-outlook-in-2027-11012307

PSMC’s current DRAM capacity mainly relies on 25nm and 38nm nodes, which restricts DDR4 production to lower-density products.

I guess that that’s more DDR4 supply drying up. It’s going to be some very scarce years for memory until enough new production comes online.

EDIT:

https://tech.yahoo.com/computing/articles/amd-ryzen-chief-teases-return-201223682.html

AMD Ryzen chief teases return of older Zen 3 chips to fight soaring RAM prices — ‘That’s something we’re actively working on right now’

Restarting production of DDR4-capable hardware isn’t going to help nearly as much if nobody is producing DDR4. I guess that there’s still DDR4 memory to scavenge from existing computers.

EDIT2:

https://www.tweaktown.com/news/109702/as-ram-crisis-intensifies-ddr3-motherboards-are-making-an-improbable-comeback/index.html

As RAM crisis intensifies, DDR3 motherboards are making an improbable comeback

Here’s one way to avoid paying an absolute fortune for RAM - forget DDR5 or even DDR4 memory, switch back to DDR3, as some folks are doing in China.

Now I regret throwing out the DDR3 memory that I have in the past.

There might be some way to make use of it.

Linux apparently can use VRAM as a swap target:

https://wiki.archlinux.org/title/Swap_on_video_RAM

So you could probably take an Nvidia H200 (141 GB memory) and set it as a high-priority swap partition, say.

Normally, a typical desktop is liable to have problems powering an H200 (600W max TDP), but that’s with all the parallel compute hardware active, and I assume that if all you’re doing is moving stuff in and out of memory, it won’t use much power, same as a typical gaming-oriented GPU.

That being said, it sounds like the route on the Arch Wiki above is using vramfs, which is a FUSE filesystem, which means that it’s running in userspace rather than kernelspace, which probably means that it will have more overhead than is really necessary.

EDIT: I think that a lot will come down to where research goes. If it turns out that someone figures out that changing the hardware (having a lot more memory, adding new operations, whatever) dramatically improves performance for AI stuff, I suspect that current hardware might get dumped sooner rather than later as datacenters shift to new hardware. Lot of unknowns there that nobody will really have the answers to yet.

EDIT2: Apparently someone made a kernel-based implementation for Nvidia cards to use the stuff directly as CPU-addressable memory, not swap.

https://github.com/magneato/pseudoscopic

In holography, a pseudoscopic image reverses depth—what was near becomes far, what was far becomes near. This driver performs the same reversal in compute architecture: GPU memory, designed to serve massively parallel workloads, now serves the CPU as directly-addressable system RAM.

Why? Because sometimes you have 16GB of HBM2 sitting idle while your neural network inference is memory-bound on the CPU side. Because sometimes constraints breed elegance. Because we can.

Pseudoscopic exposes NVIDIA Tesla/Datacenter GPU VRAM as CPU-addressable memory through Linux’s Heterogeneous Memory Management (HMM) subsystem. Not swap. Not a block device. Actual memory with struct page backing, transparent page migration, and full kernel integration.

I’d guess that that’ll probably perform substantially better.

It looks like they presently only target older cards, though.

This world is getting dumber and dumber.

Ehhh…I dunno.

Go back 20 years and we had similar articles, just about the Web, because it was new to a lot of people then.

searches

https://www.belfasttelegraph.co.uk/news/internet-killed-my-daughter/28397087.html

Internet killed my daughter

https://archive.ph/pJ8Dw

Were Simon and Natasha victims of the web?

https://archive.ph/i9syP

Predators tell children how to kill themselves

And before that, I remember video games.

It happens periodically — something new shows up, and then you’ll have people concerned about any potential harm associated with it.

https://en.wikipedia.org/wiki/Moral_panic

A moral panic, also called a social panic, is a widespread feeling of fear that some evil person or thing threatens the values, interests, or well-being of a community or society.[1][2][3] It is “the process of arousing social concern over an issue”,[4] usually elicited by moral entrepreneurs and sensational mass media coverage, and exacerbated by politicians and lawmakers.[1][4] Moral panic can give rise to new laws aimed at controlling the community.[5]

Stanley Cohen, who developed the term, states that moral panic happens when “a condition, episode, person or group of persons emerges to become defined as a threat to societal values and interests”.[6] While the issues identified may be real, the claims “exaggerate the seriousness, extent, typicality and/or inevitability of harm”.[7] Moral panics are now studied in sociology and criminology, media studies, and cultural studies.[2][8] It is often academically considered irrational (see Cohen’s model of moral panic, below).

Examples of moral panic include the belief in widespread abduction of children by predatory pedophiles[9][10][11] and belief in ritual abuse of women and children by Satanic cults.[12] Some moral panics can become embedded in standard political discourse,[2] which include concepts such as the Red Scare[13] and terrorism.[14]

Media technologies

Main article: Media panic

The advent of any new medium of communication produces anxieties among those who deem themselves as protectors of childhood and culture. Their fears are often based on a lack of knowledge as to the actual capacities or usage of the medium. Moralizing organizations, such as those motivated by religion, commonly advocate censorship, while parents remain concerned.[8][40][41]

According to media studies professor Kirsten Drotner:[42]

[E]very time a new mass medium has entered the social scene, it has spurred public debates on social and cultural norms, debates that serve to reflect, negotiate and possibly revise these very norms.… In some cases, debate of a new medium brings about – indeed changes into – heated, emotional reactions … what may be defined as a media panic.

Recent manifestations of this kind of development include cyberbullying and sexting.[8]

I’m not sure that we’re doing better than people in the past did on this sort of thing, but I’m not sure that we’re doing worse, either.

https://en.wikipedia.org/wiki/We_Didn't_Start_the_Fire

“We Didn’t Start the Fire” is a song written by American musician Billy Joel.

Joel conceived the idea for the song when he had just turned 40. He was in a recording studio and met a 21-year-old friend of Sean Lennon who said “It’s a terrible time to be 21!”. Joel replied: “Yeah, I remember when I was 21 – I thought it was an awful time and we had Vietnam, and y’know, drug problems, and civil rights problems and everything seemed to be awful”. The friend replied: “Yeah, yeah, yeah, but it’s different for you. You were a kid in the fifties and everybody knows that nothing happened in the fifties”. Joel retorted: “Wait a minute, didn’t you hear of the Korean War or the Suez Canal Crisis?” Joel later said those headlines formed the basic framework for the song.[4]

https://www.youtube.com/watch?v=eFTLKWw542g

🎵 We didn’t start the fire 🎵
🎵 It was always burning since the world’s been turning 🎵
🎵 We didn’t start the fire 🎵
🎵 No, we didn’t light it, but we tried to fight it 🎵

[email protected] looked smug as hell. They’d been telling everyone for years.

Summary created by Smart Answers AI

chuckles

And why Bash and not another shell?

I chose it for my example because I happen to use it. You could use another shell, sure.

Should we consider “throwaway” anything that supports interactive mode of your daily driver you chose in your default terminal prompt?

Interactive mode is a good case for throwaway code, but one-off scripts would also work.

The point I’m making is that bash is optimized for quickly writing throwaway code. It doesn’t matter if the code written blows up in some case other than the one you’re using. You don’t need to handle edge cases that don’t apply to the one time that you will run the code. I write lots of bash code that doesn’t handle a bunch of edge cases, because for my one-off use, that edge case doesn’t arise. Similarly, if an LLMs is generating code that misses some edge case, if it’s a situation that will never arise, and that may not be a problem.

EDIT: I think maybe that you’re misunderstanding me as saying “all bash code is throwaway”, which isn’t true. I’m just using it as an example where throwaway code is a very common, substantial use case.

I don’t know: it’s not just the outputs posing a risk, but also the tools themselves

Yeah, that’s true. Poisoning the training corpus of models is at least a potential risk. There’s a whole field of AI security stuff out there now aimed at LLM security.

it shouldn’t require additional tools, checking for such common flaws.

Well, we are using them today for human programmers, so… :-)

No problem; I remember being delighted to learn that there was a name for the thing, years back.

Also, one other comment regarding the “change away from mechanical toggle”. If you got the machine pre-built, you may never have noticed this, but on ATX motherboards, there’s a set of pins which you fit the power and reset switch wires onto.

I mean, you can plug whatever you feel like onto those pins and stick your power and reset buttons wherever you feel like, if you don’t like the position of the existing case switch. It’s just a momentary switch. You can grab replacement ones that aren’t built into a case:

https://www.amazon.com/Warmstor-2-Pack-Computer-Supply-27-inch/dp/B074XDTVN1

Or even just get your own switches and connect the plug and wires to whatever sort of momentary switch you want. Amazon or Mouser or DigiKey will have all sorts of momentary switches.

Security is where the gap shows most clearly

So, this is an area where I’m also pretty skeptical. It might be possible to address some of the security issues by making minor shifts away from a pure-LLM system. There are (conventional) security code-analysis tools out there, stuff like Coverity. Like, maybe if one says “all of the code coming out of this LLM gets rammed through a series of security-analysis tools”, you catch enough to bring the security flaws down to a tolerable level.

One item that they highlight is the problem of API keys being committed. I’d bet that there’s already software that will run on git-commit hooks that will try to red-flag those, for example. Yes, in theory an LLM could embed them into code in some sort of obfuscated form that slips through, but I bet that it’s reasonable to have heuristics that can catch most of that, that will be good-enough, and that such software isn’t terribly difficult to write.

But in general, I think that LLMs and image diffusion models are, in their present form, more useful for generating output that a human will consume than that a CPU will consume. CPUs are not tolerant of errors in programming languages. Humans often just need an approximately-right answer, to cue our brains, which itself has the right information to construct the desired mental state. An oil painting isn’t a perfect rendition of the real world, but it’s good enough, as it can hint to us what the artist wanted to convey by cuing up the appropriate information about the world that we have in our brains.

This Monet isn’t a perfect rendition of the world. But because we have knowledge in our brain about what the real world looks like, there’s enough information in the painting to cue up the right things in our head to let us construct a mental image.

Ditto for rough concept art. Similarly, a diffusion model can get an image approximately right — some errors often just aren’t all that big a deal.

But a lot of what one is producing when programming is going to be consumed by a CPU that doesn’t work the way that a human brain does. A significant error rate isn’t good enough; the CPU isn’t going to patch over flaws and errors itself using its knowledge of what the program should do.

EDIT:

I’d bet that there’s already software that will run on git-commit hooks that will try to red-flag those, for example.

Yes. Here are instructions for setting up trufflehog to run on git pre-commit hooks to do just that.

EDIT2: Though you’d need to disable this trufflehog functionality and have some out-of-band method for flagging false positives, or an LLM could learn to bypass the security-auditing code by being trained on code that overrides false positives:

Add trufflehog:ignore comments on lines with known false positives or risk-accepted findings

I keep seeing the “it’s good for prototyping” argument they post here, in real life.

There are real cases where bugs aren’t a huge deal.

Take shell scripts. Bash is designed to make it really fast to write throwaway, often one-line software that can accomplish a lot with minimal time.

Bash is not, as a programming language, very optimized for catching corner cases, or writing highly-secure code, or highly-maintainable code. The great majority of bash code that I have written is throwaway code, stuff that I will use once and not even bother to save. It doesn’t have to handle all situations or be hardened. It just has to fill that niche of code that can be written really quickly. But that doesn’t mean that it’s not valuable. I can imagine generated code with some bugs not being such a huge problem there. If it runs once and appears to work for the inputs in that particular scenario, that may be totally fine.

Or, take test code. I’m not going to spend a lot of time making test code perfect. If it fails, it’s probably not the end of the world. There are invariably cases that I won’t have written test code for. “Good enough” is often just fine there.

And it might be possible to, instead of (or in addition to) having human-written commit messages, generate descriptions of commits or something down the line for someone browsing code.

I still feel like I’m stretching, though. Like…I feel like what people are envisioning is some kind of self-improving AI software package, or just letting an LLM go and having it pump out a new version of Microsoft Office. And I’m deeply skeptical that we’re going to get there just on the back of LLMs. I think that we’re going to need more-sophisticated AI systems.

I remember working on one large, multithreaded codebase where a developer who isn’t familiar with or isn’t following the thread-safety constraints would create an absolute maintenance nightmare for others, where you’re going to spend way more time tracking down and fixing breakages induced than you saved by them not spending time coming up to speed on the constraints that their code needs to conform to. And the existing code-generation systems just aren’t really in a great position to come up to speed on those constraints. Part of what a programmer does is, when writing code, is to look at the human-language requirements, and identify that there are undefined cases and go back and clarify the requirement with the user, or use real-world knowledge to make reasonable calls. Training an LLM to map from an English-language description to code is creating a system that just doesn’t have the capability to do that sort of thing.

But, hey, we’ll see.

There was a famous bug that made it into 95 and 98, a tick counter that caused the system to crash after about a month. It was in there so long because there were so many other bugs causing stability problems that it wasn’t obvious.

I will say that classic MacOS, which is what Apple was doing at the time, was also pretty unstable. Personal computer stability really improved in the early 2000s a lot. Mac OS X came out and Microsoft shifted consumers onto a Windows-NT-based OS.

EDIT:

https://www.cnet.com/culture/windows-may-crash-after-49-7-days/

A bizarre and probably obscure bug will crash some Windows computers after about a month and a half of use.

The problem, which affects both Microsoft Windows 95 and 98 operating systems, was confirmed by the company in an alert to its users last week.

“After exactly 49.7 days of continuous operation, your Windows 95-based computer may stop responding,” Microsoft warned its users, without much further explanation. The problem is apparently caused by a timing algorithm, according to the company.

My anecdotal experience is that Vista - while pretty - is a bit of a bloatfest regardless of what hardware you run it on.

I use Linux, so I haven’t personally run into it, but is that just because of the Aero interface stuff? IIRC a lot of that can be disabled.