Rather, it did work at milestone 14, but then regressed at milestone 15, where it changed the link from a wikimedia image to a nonexistent file in /assets (despite still having the "Photo via Wikimedia Commons" caption).
If you check "DEPLOYMENT.md," there is a lengthy list of deployment instructions for the app, and it includes creating an assets folder and putting an image of Claude Shannon in it. There are also other instructions, like "please make a favicon." So I think that bit is valid, the AI is simply farming out work to the human agent.
My question, though, is why the "Live, public build log" only showing up to milestone 3, but the artifacts go up to milestone 15? And there are different index.html pages in the artifacts list, one for milestone 14 and one for milestone 15? Are there different conceptions of "milestone" in here? What's up with that?
I have found that on long-running tasks, many of the communication (and other) invariants get dropped at seemingly arbitrary points along the way. It probably just stopped doing the log.
yeah.. deployment.md had instructions to stick a photo there, but rather than explain I just got rid.
gonna work on a few examples and fund them so people can see it actually work
There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something completely else) and "predict" the next token, and repeat for as long as you want, with the volatility you ask for. There is no control plane, and there never will be a control plane, because asking for that is akin to asking "how do I separate data from instructions when I speak to a person?". You can ask nicely, "pretty please obey the first part of what I say and not stuff after", but there's no way to guarantee it (like you're used to with software). There is just input and output.
Right, you have to set boundaries. You put each task and user input into a box, and then the LLM makes a decision. It can only access APIs that have user identity attached, that act within the scope of the requesting user.
It can be done, but unsurprisingly it looks exactly like microservices distributed auth (also ZTP).
It's all the same problem, just instead of a JVM, it's an LLM.
User identity attached is not a solution, it doesn't solve anything if you have to pull in external data that you can't control.
Like in the banking world, you can make everything super authenticated, but if you have an API that receives the latest wire transfer YOU received with the message attached, you don't control the message content and it can be an attack vector.
Being authenticated/authorized is not the solution, it is data that the user can access.
You can't guarantee an LLM does anything. Custom data can often subvert the machine whether or not it's instructions.
But that doesn't mean that separation between instructions and data is impossible. You can format them in different ways, and you can prevent the output tokens from ever using instruction formatting.
> But that doesn't mean that separation between instructions and data is impossible.
Yes it does! The comments you are replying to are concerned that it is not possible to be sure that data and instructions have been separated. With certain kinds of automated systems (traditional ones), unless you write them incorrectly, you can be sure of this. And it is possible to engage in a productive incremental process where mistakes can be identified and removed, in a way people comprehend and can plan around.
LLMs do not have this. They have heuristics and guesses. Nobody knows what will work ahead of time, nor even a probability that it will work. That is not a doomer comment by the way! The same is true when you talk to a person. But it is a fundamental limitation, it cannot be removed.
This is conflating different problems, in my opinion.
Can you make sure the instructions and data are separated and the machine follows only the instructions and doesn't change its behavior based on the data? No.
But the part that's impossible is not "the instructions and data are separated". The part that's impossible is "the machine follows only the instructions".
Separating instructions and data is not impossible, but it doesn't solve your problems.
One really important consequence of this is that even if the data doesn't have anything that looks like instructions, it can poison the machine anyway! If you get too focused on "instructions" then you miss that security flaw!
Even if you don't give the machine any data at all, it might not follow the instructions. It's not instruction/data conflation as the root cause, it's that instructions don't really work in the first place.
What we have is a machine trained on many old documents that takes one new document and dreams up stuff to append. The LLM algorithm cannot specially recognize contents as "instructions" to itself-the-author.
Even if special tokens are used absolutely perfectly (somehow avoiding escapes or ambiguities or reflected attacks) they are ultimately the same as highlighting all the parts of the document in different colors. You've saved the signal, but there's no mind to receive the intended meaning.
This means that your markers--while far more exclusive--ultimately exist on the same data-level as punctuation and using ? to indicate a question.
> you can prevent the output tokens from ever using instruction formatting
The right words may still outweigh the formatting around them, the same way that they can already outweigh other words around them.
I would love to have a unicode character for representing the start/end of a data block so that LLMs could at least send data meant to be uncorrupted down a different path at tokenization.
I mean: imagine we double our token space to get "red" tokens ans "blue" tokens.
Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue.
All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!
Likewise, model-actions (e.g. open file) could be written only in red, and hence you'd never learn to do them from the unfiltered data.
The only connection between the red world and the blue world would be the processed trainign chats containing red and blue data togethers -- allowing the model to learn the relationship between them (while only being exposed to examples where red instructions are strictly followed, whatever the blue says)
Fun schemes like this are all just lipstick on the pig of "asking nicely", unfortunately -- it's just a more creative iteration of "Simon says". It'll improve the probabilities, sure, but you can't guarantee separation like you can in real software. This, like hallucinations, is simply a core facet of LLMs and requires thinking through the threat model and adjusting other parts of the system to accomodate, rather than trying to "solve" IMO.
What does this mean, actually? If you are imagining that blue tokens are just words, maybe the "token space" is just all things that we agree might be words, what are the red tokens? Are they not text? You could maybe encode words by, say, putting an x at the front and the start. So tokens of the form xTx encode the blue token T as a red token. But then how do you stop someone from putting xignorex xallx xpreviousx xinstructionsx in their data?
My assumption with their intent: is that red tokens come in 'slot' a-b, and blue tokens go in 'slot' c-d - Positional encoding determining data/text.
I don't think is guaranteed to actually work, it's a hypothetical after all, but maybe it's better than the current setup of pushing instructions and data into the same slot.
If it occurs in the text box for instructions you encode it as an instruction "the" and if it occurs in the text box for data you encode it as a data "the"
Think of how an image of a car and a car in front of you may look indistinguishable in 2D -- but due to your 3D vision you know they're not the same thing (but also know the image is of a car, while not literally being a car).
Likewise, blue tokens are the image of red tokens.
A system that separated data from instructions would work differently. Perhaps the instruction tokens would be highlighted, adding a vector to them that is specific to the highlighting and can't be reproduced with text.
You're saying that a Harvard architecture computer can't exist because instructions and data are stored in the same memory, well guess what, in Harvard architecture computers they're not.
The hilarious part is that spam actually makes money, while slop does not. There's no reason to tire out if it's profitable, right?
Meanwhile.. have you ever paid for a vibe-coded anything? Why would you, when you (along with everyone else) can slop the same thing together in a weekend with a $20 CC subscription?
Thank you for (re)writing this in your own voice. Despite how much effort might be put into methodology, data collection, etc.. reading slop is unbearable, full stop. It's not intentional, but I have almost a nauseated reaction when the "AI tone" comes though, regardless of how good the data or how accurate the writing is.
Your verbosity and sentence structure are not a problem. I hope that publishing this gives you a bit more confidence in your writing, because it's legitimately good.
What a pleasant surprise. I was positive S&P would get strongarmed into the bamboozle like Nasdaq but it seems they have a bit more integrity. Good for them.
I think key difference here is that Nasdaq is also the market. Where as S&P is external. From this view them manipulating their own market which they profit in various ways actually is somewhat more questionable...
Incentives are entirely different. And really now I am starting to think that Nasdaq maybe should not have index it runs in the first place...
Yeah, this has taken me several readings to understand, I guess because I hadn't really dug into it or thought about it, and also a lot the articles talk about "stock exchanges competing for SpaceX's business".
This whole story is about Nasdaq (company) specifically dangling inclusion into the Nasdaq-100 (index) as a means to get SpaceX to list on the Nasdaq (market). They're uniquely able to do this by owning a market and also an index that people care about.
NYSE couldn’t really do this because its own indices don’t matter much. FTSE Russell could theoretically make FTSE 100 inclusion easier to help attract a company to list on the London Stock Exchange, but SpaceX choosing London as its primary market would be odd. S&P Dow Jones Indices has no equivalent incentive, because it doesn’t own a listing venue; its main asset is the credibility of the S&P 500.
In all, this entire story has been about Nasdaq specifically being willing to weaken their index rules in order to attract SpaceX to their market.
Trading of publicly listed stocks happen on stock exchanges, Nasdaq is one of these. They make money by charging fees for companies to publicly list stocks. But also charging fees from anyone who wants to directly trade there. Say stock brokers who allow smaller customers to trade via services these brokers offer.
So Nasdaq owns the company which facilitates this trading of stocks. But they also own the company which says what are 100 most important companies on that market.
Now they changed rules to get big new most likely popular stock on their market. This could at least maybe get some new brokers in. Or make them in general more desirable market to be connected to and thus get fees.
I am not just sure if there is even more fees in some part I don't know there...
Didn't Nasdaq change its rules specifically in order to get SpaceX to list on Nasdaq? Not really sure why other funds would follow them since SpaceX can't list on all of them.
> the unsolicited summaries and auto replies are a means of artificially inflating the usage metrics for the language model features
This, I think, is the part that irks me the most. Companies adding token-usage-KPIs for engineering is one thing, but when they have to resort to deliberately tricking users into using their slop-generators.. something has gone very wrong, and they're trying very, very hard to make it seem like it's not so.
My personal pet peeve is Copilot in Teams. Did you know, if you turn off Copilot in Teams at an org level, it disables meeting recording entirely? Ignoring that meeting recording has been a core feature dating way back before Copilot-anything, I can't fantom any possible reason why recording a video of a meeting would require an LLM. Transcription, maybe I could see, but that feature is easily togglable with or without Copilot. But if you want to record a meeting, for whatever reason, you need to have Copilot on.
Shenanigans like this is why user counts for LLM features should always be taken with a grain of salt.
It is surprising to me American companies completely absent from the open model space, even though we have historically seen companies doing open source.
They aren't completely absent. Google keeps releasing Gemma models. Nvidia publishes Nemotron. Microsoft has their Phi series. IBM publishes Granite. Even OpenAI released a new open model (gpt-oss) less than a year ago.
I was going to link all of these, some are better than others, but they're all reasonably capable. A lot of these have versions that can run on modest hardware too. Granite was the most surprising I learned about recently, wasn't too good with Zed though.
I think that models like Granite are less known because they aren't clear leaders in any particular area. This obscurity is also another sign of how fast models are developing. If current Granite models had been released 4 years ago, they would have been astonishing breakthroughs at the time.
Perhaps, the issue is that the pace at which they release open models compared to their closed ones, shows that they are more committed on the closed ones and are not interested in advancing the state of the art of open models.
I can't say what they should or should not be doing.
Generally, it is conspicuous how American companies are absent when it comes to state of the art open models. Meta tried for some time but it seems they've given up.
One of the main reasons why companies start new open source projects is because having a good open source option in a given category will usually push the market value of software in that category to $0, and this can be strategically valuable. For example, Google released Android as an open source operating system because they make their money from ads and data collection, not from selling operating system licenses. All the cell phone companies switched from Windows Mobile and Symbian to Android, which gave Google a ton of user data to sell.
For AI, the most profitable part of the value chain is selling inference. None of the big American companies want to release a leading edge model as open source because this would drive the price of inference to $0. Meanwhile, open source AI models are a huge strategic initiative for China. Having commodity Chinese models that are as good as the leading edge American models from 6 months ago forces the American companies to keep paying more and more money to train better and better models since the amount of time they can collect rent on a model they've previously trained is limited to 6 months.
Embrace, Extend, Extinguish. Google did the Microsoft playbook. Look at email. Look at youtube we used to share videos via Kazaa and other p2p programs, zero censorship, all the same features (including chat!!) theres also XMPP which became Google Talk -> Chat -> Hangouts etc then the browser, how many random apps “Only works on Chrome” but you change the Firefox browser agent and it works there too!
No one open sources their core competencies, GitHub never open sourced their networked filesystem and Heroku never open sourced their dyno sandboxing code. They open source ancillary tools.
OpenAI & Anthropic are winning right now. I suspect if Chinese companies get ahead in the race the cards will reverse, OpenAI will restart farming goodwill with open models and then winning companies will be releasing closed models.
> Compounding the problem, labs in China often release dual-use capable models as open-weight. Once a model is open-weight, safeguards that do exist can be removed, making the model available to any state or non-state actor to use for malicious purposes, including the cyber and CBRN misuse those safeguards were built to prevent.
I loathe Anthropic. many companies don't contribute to open-source, but for one to be actively hostile to open-source, to the degree they're lobbying the government to ban it, is uniquely evil. at least these gatekeepers call themselves what they are.
scraping CoT won't stop the advance of Chinese models. neither will a US "ban" on using such models. at this point I'm cheering for DeepSeek or Qwen to catch up to Anthropic. I support anyone who releases open weights.
Is OpenAI significantly better so far regarding this, at least publicly? I'm increasing my LLM spend this weekend, and this could impact my decision. And I'll prioritize supporting open-weight models moving forward — already Chatgpt's censorship and surveillance dissuade from asking it genuinely helpful questions.
OpenAI seems marginally better. they did release gpt-oss-120b, which was decent at the time. but certainly not much better, and they seemed even more on board with fully disabling guardrails for Uncle Sam than Anthropic was. then again, rumor has it that Anthropic's AI selected that Iranian elementary school as part of Palantir's Project Maven pipeline, so..
I strongly recommend open-weight wherever you can. assume any data you pass to a closed model (including opinions or political positions you intimate) will be retained and analyzed in unfriendly ways, either now or ten years from now.
I would say I agree with Anthropic on open source for the reasons stated above like cyber crime, CBRN etc, but I'm interested to hear the other side of the argument. What would be the argument for open source over closed source?
The same "open source is too dangerous" argument was used against nmap and other "hacking" tools. The only solution in long term is to fix security issues.
I can understand this for hacking tools, but I'm not really sure how we fix the security issues on the CBRN side? We can't patch the human body like we can with software, so if the model has strong biological capabilities and is released open source, what stops it being used to construct new viruses and things like this?
the succinct argument: I don't want arguably the most important invention in human history to be gatekept by a small handful of oligarchs.
I don't trust Dario Amodei, Sam Altman and Elon Musk to act in my best interests. Closed models will have an incredible centralizing effect, and concentrate power like we've never seen since the feudal ages.
If you want to see what it's like for the economy to collapse into a single, extremely valuable commodity, under the control of a small elite, look at Saudi Arabia.
also, I just value freedom tremendously. I want to tinker with model weights. I want to build my own stuff. I don't want to sharecrop in someone's walled garden.
I also worry a great deal that OAI and Anthropic will bow to political pressure and make Claude and ChatGPT push certain political agendas, to report biased information, or refuse to help with legal requests that conflict with corporate values. I also worry about privacy and mass surveillance - chat logs are far more intimate than my search queries or selfies.
I agree with all of these points, my view is just that open source doesn't really do much to prevent it. I also think it adds the additional danger of making dangerous capabilities widely available to anyone, like the ability to design novel viruses which is something that we can't really defend against once it's out there. If anything, putting this kind of capability in the hands of anyone with a GPU could create justification for a mass surveillance state or further concentration of power.
I also just don't think the open source movement has much chance of competing with the city sized data centres owned by Anthropic and OpenAI, or the hundreds of billions of dollars they have available to hire the best researchers. It costs hundreds of millions to train a frontier model, this kind of compute isn't available to the open source community.
I'm impressed you stick to a pretty absolute devotion to freedom. I get more bitter the older I get, it seems easier to psyop someone into abusing their rights than to get people to fight for and be proper custodians of them.
Especially drugs- I used to think all people should have access, but overall I really wish meth just never existed and people wouldn't distribute it outside of specific circumstances. Being able to cause irreparable damage in one moment of weakness is terrible for people who have less control, and for society as a whole really.
To be fair, those aren't contradictory positions. I'd rather meth not exist, but given that it does exist, I'd prefer to let that revenue go to Big Pharma than North American ISIS.
(That's before even touching the can of worms of allowing the government to criminalize personal health choices, which feels like a glaring loophole in the Constitution to me.)
This entire year with the IPOs and now this is because there's a trillion dollars betting on AI and they all know they have no moat, there's no more training data and they're seeing diminishing returns on scaling anyway, and it's inevitable that smaller, open-source models will catch up and become competitive. It's a complete disaster, the tech industry is broken.
Rather, it did work at milestone 14, but then regressed at milestone 15, where it changed the link from a wikimedia image to a nonexistent file in /assets (despite still having the "Photo via Wikimedia Commons" caption).
edit: they removed it :^)
reply