The arXiv paper was submitted April 2025, the research itself isn't new, but the new is Google's blog post packaging it for a wider audience.
worth reading the original paper alongside the blog post. I think the ppaper has details the blog post glosses over, particularly around the calibration-free quantization approach and how they handle outlier channels.
Interestingly: the research sits on arXiv for a year, nobody talks about it
boilerplate overlap is expected, no one rewrites these from scratch
though when everything lines up the same way across hundreds of reports, it gets weird...
I mean look at those reports; same pagination, same auditor showing up almost everywhere, no exceptions across all clients.. not even efficient templates should be like that
you still expect variation in scope, findings, structure, even if the base language is reused
yes. I think some overlap is normal, but this is not that, eg. seen:
• same pagination across hundreds of reports → 100% template output
• same auditor license everywhere → either extreme concentration or just rubber stamping
• zero exceptions across all clients → unrealistic, real audits always find something.. right?
• system descriptions pulled from marketing sites → .. copy paste
at one point you’re really looking at reports that were never really produced per each company
fair call on the popups, they’re not real-time. I added them quickly to make the page feel less empty while testing engagement, probably not the best call in hindsight and I’ll remove or replace them with something real.
on the "vibecoded" part, yeah I moved fast. this was built in under a day to get something out and see if people even care about this angle. that doesn’t mean the underlying data or direction is fake though.
the domain choice is just speed and availability, not some SEO master plan. if this turns into something real I’ll move it to a proper brand/domain.
and yeah I get why it looks like a growth/SEO play, but the actual goal is to push more transparency around these audits. if I just wanted traffic there are easier angles than going after something this niche and messy.
either way, appreciate you calling it out, some of it is fair and already being fixed.
Cybersecurity compliance, but .xyz site built under a day to get something out fast to drum up engagement and "test the vibes on the idea". Makes one wonder what became of this industry.
hello, this isn’t a competitor. I run a consulting company in the cybersecurity space and saw a chance to make this whole process more transparent.
I agree it came off a bit clickbaity, I'm sorry, Claude probably pushed it too far. but I don’t have an audience anywhere, no following on social, so I needed to ship something fast and make it engaging. the intent wasn’t just this Delve thing, the goal is to move away from it and turn it into a proper hub for compliance transparency over time. But i need a way to marketing this intially.
it’s been less than 24h, I built and pushed everything pretty quickly, so yeah there are rough edges. I’m already working through them and fixing things.
on the account being new, I get how that looks. I mostly use X and reddit, this is actually my first time posting on HN so I had to create an account.
Not clickbaity, but downright misleading and, for all we know about the XHR traffic, potentially malicious.
> But i need a way to marketing this intially.
Posting a site which blatantly fakes statistics, such as the ones mentioned two posts up from here, is not doing any favors.
Sending XHR requests with opaque binary data every few seconds is not doing you any favors. No information-only site has _any_ business XHR-posting opaque state every several seconds.
"Post it quickly at all costs, accuracy be damned," is not a viable strategy for a legitimate and believable site.
I'd find this more compelling if you looked at a few thousand Vanta or Drata reports grouped by auditor. You're going to find the same commonalities with only trivial language differences.
SOC2 reports are private between you and the auditor (that way if you "fail" you can just find another auditor or have a re-do, and no one is the wiser), and basically always gated behind a sales touchpoint (another hint about what utility they provide). I guess the Delve ones leaked which is why they can all be compared.
220 out of 494 "no exceptions" seems quite high to me. Nobody I've ever dealt with allows an exception to make its way into the report.
"You" probably don't, but it's not just "you". There's also the counterparty who's asking to see that report. Maybe they're doing it for paper-pushing purposes of their own, but ultimately, somewhere up the chain, there's someone thinking "I can't personally audit all my suppliers, and I can't be sure they're doing the right thing, so I'm going to ask them to get an independent audit".
Of course, this shows that the entire system is a bit of a charade, but the point is that someone cares and they're gonna be annoyed when they find out that the audit appears to be a sham.
Whether they have a good alternative is a separate question. But here's another way to look at it: if we show blatant disregard for self-regulation, the government is eventually going to show up and come up with more onerous rules.
Is it true, though? Or has everyone just been psyched into asking for that certification out of a vague fear of "consequences" or of being left behind?
It's not either-or. Companies care about security because of the consequences. If you're a big company contracting a small one, you don't want to get owned through that vendor because you know you'll be the one holding the bag (data loss, reputational damage, regulatory scrutiny, lawsuits).
Small vendors will tell you what you want to hear because they're desperate for your business. Independent auditing is, in theory, a way to get closer to the ground truth. Well, in theory.
Probably not, in fact your auditors not being terribly thorough might be a selling point. But your clients, who are the ones asking for the box to be checked, might.
In my experience, clients don't dig deeply into the report or the auditor, they just want to see that you 1) have the report 2) it doesn’t have any egregious exceptions. Perhaps if this makes big enough news, that’ll change.
As the company? No. In fact, it's likely better for you if they do a bad job. You potentially get shielded from blame, but don't actually have to put in the work.
As a user/customer/potential victim? Yeah, you do.
Most of this isn't that damning. SOC IIs are already highly templatized, so pages matching up really isn't meaningful. In fact, an overly detailed or overly verbose template is more likely to have matching pages since you'd never have to add additional content to it.
System descriptions don't necessarily hold much weight. They're often more about giving a general shape of the system to help orient the reader, rather than providing a technically complete picture.
Most of the meat in these is about the controls being tested (which are semi-standardized within an auditor) and the results. Many of these controls are really basic and easy to get "no exceptions noted".
That being said, nearly everyone has at least one exception, even if it's minor. The fact that they didn't find any across all of their clients is a strong indicator they're not diving deeply enough.
The no exceptions noted piece is kinda funny. Most SOC2 auditors at least put in the minimal effort of finding one person who didn’t do their cybersecurity training, so the report’s not total boiler plate.
