Not sure how I feel. Anna’s Archive turned into a profit-seeking beast a long time ago. They’re also rolling in it thanks to he massive deals to “license” the content to AI companies.
Fred, nice to meet you. The grading model had no idea what was being tested. We used separate accounts to compartmentalize. The Claude grader was guessing GPT-3.5 Turbo or GPT-4 by the end. On the coding block it consistently scored responses as GPT-4o level. We followed the MT-Bench grading guidelines as published by the team that created them. Did the research, followed the book, had no horse in the race. Every score and every response is published in the tape so you can regrade the whole thing yourself if you want. And this is me typing, I'm just a guy in LA who spent a weekend running 80 questions through a 2B model and thought the results were interesting enough to share.
Seems to be llm written article and the tooling around the model is undeniably influenced by knowledge of the tests.
In all cases, GPT 3.5 isn’t a good benchmark for most serious uses and was considered to be pretty stupid, though I understand that isn’t the point of the article.
really appreciate you reading the article. the benchmark data, grading, and error classes were all done by hand though. the ~8.0 is the raw model with zero tooling, and the guardrail projections are documented separately. and yeah gpt-3.5 isn't the gold standard anymore, we're on the same page there. we just wanted to show that the quality people are still paying for can be free, private, and customized to whatever you need. thanks again for taking the time to check it out.
We actually just ported SecureStore to go, it’s sort of like this but with cross platform clis and intended to also allow sharing secrets across services and languages, in a secure and embedded fashion! It’s available in rust, php, .net, JS/TS, Python, and golang and easy to port to others.
The approach to crypto is very different, we went with what’s very well understood and very well supported on all platforms with little or no dependencies (eg we can use web crypto in JS frontend or backend with no external libs or crypto JS library nonsense).
The original .NET and Rust code is from over a decade ago and carefully architected (well before vibecoding was a thing), the secrets are stored in a human readable (json) vault that can be embedded in your binaries or distributed alongside them and be decrypted with either password-based or key-based decryption (or both).
That’s actually a pretty interesting tradeoff — especially going with “boring crypto” that’s widely supported vs pulling in heavier deps.
The JSON vault + cross-language portability is nice too, especially if you’re embedding secrets across services without tying yourself to one runtime. Curious how you handle key management at scale though — that’s usually where these systems get tricky more than the crypto itself.
SecureStore is an open spec/protocol for managing secrets in a secure and portable manner, while it defines the decryption key formats (currently: key-based, password-based, or a mix of both interchangeably) it doesn't get into the mechanics of key management, which are "trivial and left as an exercise for the reader."
More seriously though, you're supposed to use separate vaults (with the same keys, where "keys" is the name of the secrets, not the decryption keys) for testing/staging/production, e.g. perhaps secrets.{testing,production,staging}.json and the same secrets.{testing,production,staging}.key for the decryption keys, and store both the username and password in them (after all, it's just an encrypted, glorified KV store) so that you don't have to hard-code any usernames and conditionally load them based on the environment in your code (so db:username is one "secret" and db:password is another (actual) secret).
The secrets vaults (the secrets.json files) are non-sensitive and can be versioned and pushed to your server the same way you push the binaries. Now how you move the secrets to the server is up to you. You could do it the old-fashioned way and just have it as an environment variable, in which case even when your env vars leak at least you haven't leaked your api keys, only the key to decrypt them (which you'd then rotate), but that's not a recommended option. Ideally you'd instead use whatever secure channel you use to init/stage the servers to begin with to transfer the secure key files - the key files are generally immutable, even as the secrets change, so you only have to do this once (ideally via a high-friction, high-auth mechanism, for most people not at FAANG scale, probably manually).
You can also use whatever additional layer of abstraction on top of the symmetric SecureStore decryption key you like. For example, you could asymmetrically encrypt the keyfiles and then each host would decrypt it with its own private key, or have a secrets side channel that's just used to obtain the static decryption key over the local network, or use your operating system's encryption facilities to transmit it, whatever works for you at whatever point on the complexity/security curve you desire.
(These are all just options, none are official recommendations.)
It’s become neigh impossible to get your own code signing cert these days. The 2025 update from the CA forum required code signing certs to be short lived (no more three or five year certs) and stored exclusively on an HSM. As a result, most companies cross-signing these certs have moved to a subscription PaaS model where you are issued a cert but never receive custody of it, and perform signing via their APIs, and are at their mercy should they decide to block your account.
Anyway, even if you could get your own cert it would be same thing: MS could revoke or blacklist your indicate cert (though usually the grounds for doing so are much less shaky than your account being suspended for vague “tos violations”)
I was afraid of the HSM at first but for an open source developer (rather than a big company) I found it wasn't a big deal. I can't sign in GitHub Actions and I have a USB stick that lights up when I sign releases, but it hasn't been a blocker. I got mine from Sectigo Store. This isn't hypothetical, I really did it, I've got the HSM, it works. It wasn't difficult. It just cost some money and a little bit of time. "Nigh impossible" is a tremendous exaggeration. I'll concede "annoying and expensive" perhaps. If you've got the money, you can get the HSM. You don't have to re-buy the HSM when you renew your certificate.
The Microsoft Store account was painful to set up, I'll note. My developer account had also been cancelled by Microsoft for unknown reasons, and I ultimately had to set up a brand new one. New email, new name. My new account has my middle initial because I couldn't clash with the existing, closed account. My first and last name alone are banished forever from the store.
The "same thing", as you concede, isn't the same thing. Quantity has a quality of its own: one happens all the time and we're reading an article about it happening right now. In the comments there's another prominent maintainer who it happened to, and it happened to me personally! That's three right here! The other happens so infrequently that people in this same HN thread are complaining that it isn't happening enough. Can you find an example that's like Veracrypt and WireGuard? In practice, it seems they rarely do this, even when they should. You can actually view the list under "Manage computer certificates" > "Untrusted Certificates." On my computer the entire list is 20 certificates.
I'm standing by my suggestion, 100%. These aren't equivalent risks at all.
Thanks for sharing your experience. I have been code signing releases for over a decade as an indie publisher myself, until I found myself effectively iced out by the HSM requirement, the increased cost, and the shortened cert lifetimes, which, as someone with certain executive order dysfunctions, I already had a hard time being on top of with the old (multi-year) lifetimes.
I just migrated to MS artifact signing and, thank the lord, had an actually easier time getting verified than I did with the Sectigo and Comodo in the past. I’m sure I’m not representative of anyone else’s experience but having already had a developer account (with a different email and without an Azure account!) that I had already been using for the Microsoft Store might have helped, as well as the fact that I had a well-established business history (I’ve heard businesses younger than 3 years can’t get verified??), but reading all the comments here makes me very uneasy about the future.
It’s good to know the HSM route isn’t a complete non-starter. The main reason I panned it is that when I started looking into this I found that a number of companies that had previously offered the HSM route had done a bait and switch and were now keeping custody unless you were big enterprise (meaning willing to put up with 10k/yr fees). I did find a few that would allow OSS devs to sign their work, but read horror stories on Reddit and elsewhere about their freezing the account and issuing no refunds if you ask them to issue the cert in the name of your LLC or corporation instead of with your personal name (which I expressly did not want). Also, they actually were more expensive than Azure artifact signing even after the HSM cost was taken out.
I believe you. I also found that many CAs will not deal with a solo developer; that's real. But Sectigo continues to offer HSMs to solo developers. The link I used is [1], you buy the HSM along with your first certificate and they ship it to you. $300/year for the cert, $90 one-time for the HSM. That's not cheap but I think for specific developers looking for an escape from the store, it's a good price for freedom. The HSM is a USB stick with an LED on the back. The software is called "SafeNet Authentication Client" and it sets up the certificate access in your Windows Certificate Store so that signtool can use it. Prompts for the password every time (annoying).
For comparison, my code signing cert via Azure (no Microsoft store account required, can be used to self-publish binaries/installers the old fashion way) is $10/month, or about a third of the price Sectigo is charging you. I figured it was worth trying this route first, though I had to write my own basic tooling around it.
I must say your experience is interesting. I am using https://signmycode.com/sectigo-code-signing, but I have chosen Install on Existing Token (Google Cloud KMS), and it's quite easy for me to handle the stuff. I am not scared of key storage or security issue nor password protection or forget issue.
That’s not how any of this works. There are separate teams within (each division of) Microsoft that could easily pull the plug on your account (or if not the entire account then your account’s access to the specific service or family of services) for any of a myriad purported reasons or alleged ToS violations.
No one is calling an executive meeting to discuss banning an OSS dev’s account.
I have mixed feelings about this. It's absolutely phenomenal that such a treasure trove was unlocked thanks to AI, but presenting the AI results are "definitive" (even with an "edit" or "report" feature that's applied equally to human-located and AI-located results) isn't really a win. The old dataset might have been incomplete, but where locations were determined, they were a result of a (probably neural/autistic/ocd) human contributor that had some measure of true confidence in the results. AI contributions are great, but imho they should never be allowed to freely mix with and dilute human contributions: the resulting dataset is permanently polluted.
Ideally they'd always carry an "AI-generated" flag (in the db and in the frontend) until manually reviewed (or never) by a human. If anything, this is actually in AI proponent's favor as it would let you periodically regenerate or cross-validate (a subset of) the AI contributions some years down the line when newer and better models are released!
(author here) Just to be clear, none of the photos were ever human-located. The system this replaced was, roughly, regular expression + Google Maps geocoding API. The only photos located by hand were the ~200 I used for my test set: https://github.com/danvk/oldnyc/blob/master/data/geocode/out...
There's actually no guarantee that if the "experiment" were allowed to continue that the results would have been as great. If the biomass accumulated faster than it could be broken down, we might not have seen the same result.
reply