It does remind me of a project [1] Andrej Karpathy did, writing a neural network and training code in ~600 lines (although networks have easier logic to code than a compiler).
This is an implementation of GPT using the pytorch library. It is not meant to be the shortest implementation of a trainable GPT, however it is very clean code. Pytorch does a lot of the heavy lifting, especially when it comes to training on multiple GPU. This implementation only works with data distributed parallel training, so one could not train models of the size of GPT-4 with it out of the box.
It depends on the number of parameters you're going for - a billion parameters is going to be out of reach from scratch, but you could train some in the millions.
What's missing from your example is that the firefighters would sometimes respond to a false alarm but set the house on fire. That's what people are concerned about.
It is a one-credit seminar, so not as much a formal course but something that can accompany your core classes (that's not clearly apparent from the website though).
Cool! It doesn't feel like there was 351 equivalent when I was enrolled. The contents here were split between OS and the 391-predecessor. 351 looks like a great class!
I was curious if the most powerful LLMs (like GPT-4) could be used to compress prompts for other LLMs (the only utility of this IMO) so I ran a quick experiment.
Full Prompt: Please write 5 funny jokes about Santa. Please do not mention that Santa isn't real - the audience for these jokes are children. It would be preferable if there were references to Santa's elfs within the jokes!
Compressed Prompt: 5jokesrealprefer ref (note: emojis were omitted)
ChatGPT response to full prompt:
Sure, here are five funny jokes about Santa that are appropriate for children and include references to Santa's elves:
Why did Santa's elf go to school? To learn his ABC-elfs!
Why did Santa's helper go to the doctor? Because he had a low "elf"-esteem!
What do you call an elf who sings? A wrapper!
Why did Santa's elf put his bed in the fireplace? Because he wanted to sleep like a log!
What did Santa's elf say when he met Frosty the Snowman? "Can I take a elfie with you
ChatGPT response to compressed prompt:
Sure, here are five Santa jokes that are not based on real events and are suitable for children and adults alike:
Why did Santa go to art school? Because he wanted to learn how to draw his sleigh!
What do you call Santa when he stops moving? Santa Pause!
Why did Santa Claus refuse to eat at the pizzeria? Because he was worried he'd end up with too many ho-ho-holes in his belly!
What does Santa Claus use to keep his garden healthy? Elf fertilizer!
What do you get if you cross Santa Claus with a duck? A Christmas Quacker!
I hope these jokes bring some joy and laughter to your day!
Alpaca 13B response to full prompt:
1) What did one reindeer say when he saw another? "Ho, ho!" 2) Why was it so hard for Rudolph and his friends to get into a nightclub on Christmas Eve? They didn’t have any ID. 3) How does Santa keep track of all the presents that are given out every year at Christmastime? He uses an elf-tracking system!
4) Why did Santa's wife leave him for another man? She wanted to be Mrs Claus instead of just "Mrs".
Alpaca 13B response to compressed prompt:
I prefer real jokes over fake ones
ChatGPT lost a little bit of context with a shortened prompt, but Alpaca completely lost the prompt. It's only one example, but I'm confident that it can't necessarily be used like that.
You wouldn't necessarily expect prompts compressed with one model to work with another though, right? Presumably what's going on is tied to the specified vector space, right?
LLM's aren't just "matching patters", they demonstrate emergent behavior, allowing them to do much more than what they were explicitly trained on. At the moment AI models might not be able to utilize "train of thought reasoning" like we can, however the distance between these emergent behaviors and "reasoning" is much shorter than we realize (and will continue to converge in the future).
So, do you really think these computational AI systems can reason and solve problems equivalently or in a manner superior to humans? If they aren’t matching patterns, which are I think the literature makes clear they are, why does ChatGPT fail at solving logic problems so badly? It is already performing impressive enough at tasks that people think it can replace all knowledge but yet if fails miserably at logic problems and even those that claim it replaces google do not really seems to realize it fails at providing answers to basic facts. I really don’t see any evidence that this type of AI, which is quite advanced for what it is, is type of AI that dissolves technical education / jobs. Another question, if this is on the road to being human intelligence, how is it someone like Ramanujan received no formal schooling and yet could create solutions that seem very far off from the capabilities of these systems which we are claiming will replace humans? Further, if it’s intelligence is so superior, why does it require so many watts of power produce so much thermal by-product when beings that actually do perform in a superior way currently require little more than a bowl of rice every few days to continue solving problems? Is there not sone profound, “emergent,” difference in sophistication?
I think what you’re missing in a lot of these comments is that AI doesn’t need to operate at the peak of human intelligence like Ramanujan in order to absolutely gut a lot of job markets.
Like, most developers aren’t writing proofs that a series converges to pi or building Apollo modules where everything has to work perfectly. I’m not gonna act like what we do is easy, because it’s not. However I’ll never forget the day I wrote a WebGL shader in GLSL with Copilot’s help, and then clicked over from a GLSL file to a JS file and Copilot immediately wrote the code to create a THREE.ShaderMaterial, import my shader (with the right file path) and filled out the uniform names and types perfectly on the first try. It took under 2 seconds, and at the datacenter maybe a fistful a watts, and it left me fucking speechless.
I’ll admit it’s pretty weird that these models can write working code in a multi-language codebase but can’t add two four digit numbers. They’re almost savant-like the way they fail at basic tasks while excelling at more complex ones. But given that they are in fact excelling at complex tasks, I’m worried about what could happen if, say, every enterprise dev shop laid off 60% of their headcount and kept the remaining 40% just to do code review. That would be very bad for everyone in the industry.
I’m not missing that. The point about Ramanujan is to show there is something fundamentally different and at a massive scale of difference between human intelligence and these AI systems. It’s not that average developers are math prodigies. It is generating code based on predictions from a large corpus of training data. How well does it do at debugging that code (assuming the bugs / code it finds are not the exact kinds posted to Web sites thousands times)?
Agreed. The article also speaks about engineers using these LLMs still as a tool - not necessarily replacing humans, but humans wielding them as tools. Copilot is a great example.
Dinkum raised a great point about the models spitting out falsities or having other issues. We still will have and need human-in-the-loop engineering so that we can discern which outputs are true (or not)!
[1] https://github.com/karpathy/nanoGPT