The amusing thing is that it takes several orders of magnitude less data to bring up a human to reasonably competent adulthood, which means that there is something fundamentally flawed in the brute-force approach to training LLMs, if the goal is to get to human-equivalent competency.
Also the fact that 30B models, while less capable than 300B+ models, are not quite one whole order of magnitude less capable, suggests that all things being equal, capability scales sub-linearly to parameter count. It's even more flagrant with 4B models, honestly. The fact that those are serviceable at all is kind of amazing.
Both factors add up to the hunch that a point of diminishing returns must soon be met, if it hasn't already. But as long as no one asks where all the money went I suppose we can keep going for a while still. Just a few more trillions bro, we're so close.
I suspect there’s a good deal baked into a human brain we’re not fully aware of. So babies aren’t starting from zero, they have billions of years of evolution to bootstrap from.
For example, language might not be baked in, but the software for quickly learning human languages certainly is.
To take a simple example, spiders aren’t taught by their mothers how to spin webs. They do it instinctively. So that means the software for spinning a web is in the spider’s DNA.
I don't think we can even do a "Handbook of LLM techniques" at this point and have something thick enough to raise a monitor. It's all in the Data. First they started with copyrighted material, then public sources (maybe private sources as well), and now they are hammering every server in existence to get moooorre..
I feel like scientists should be explaining to us how the world is, and then other people should use those explanations to try and improve it.
Right now I feel like there are a scientists who would hide or discard results if they contradicted their advocacy beliefs,which is a dangerous place to be imo.
That's how it works. I think people for some reason don't understand policy making. The CDC conducts research and studies, pulls data, performs analysis, and then provides guidance based on it.
It enacts no rules, laws, or regulations. That's done by policy makers who can listen to or ignore the guidance and data from the CDC at their discretion.
No, it's not. Good science requires objective thinking and evidence-based reasoning. Claims must be proven, not accepted based on authority or prima facie evidence.
Unfortunately, social media or whatever has changed science communication. A scientist can do amazing science, have total evidence based reasoning and then be completely ignored because some quack tells a good story on Rogan.
That doesn’t matter and it’s not any different than 20 years ago. New findings could be published in medical journals that nobody in the public hears about and some quack on Howard Stern could spew to millions.
You might be confusing activists with volunteers. Those who donate their time and money are not necessarily vocal about their pet projects. I'm part of the latter and do not consider myself an activist.
Absolutely not. A scientist is expected to change their mind when new, counterfactual evidence is presented. Activists push for positions regardless of whether any evidence exists to support their position, and seem to maintain their position even when presented with counterfactual evidence. That is not science. We have a name for it: dogma.
Wait a second, if you’re saying that this is not a feature of good activism, are you implying you are more convinced by activists who practice dogma over objective thinking?
What is a scientist to do when they discover a vaccine or cure for something; say fuck it who cares if we change behavior? Are you saying a good vaccine advocate is someone who ignores the underlying science and acts dogmatically?
It just feels like you want to demonize this action of activism for… why? Just because there are lots of bad activists? There are a lot of bad scientists as well, to be honest the view of “good scientist” and “bad activist” feels dogmatic.
I have yet to observe an activist practice objective thinking. That was the root of the argument. Activists sometimes do back the correct argument, but not because they are practising scientific reasoning. Most activists are swayed by rhetoric, a good story. That's an emotional response, not a logical one.
To answer your second point, science has a process for disseminating new findings. It's not perfect, but it works. Organizations that scientists work for do pay attention to those sources, discoveries do get patented and productionized. I encourage you to conduct some research: See how many people were talking about mRNA vaccines and gain-of-function research on social media before COVID vs after. The lack of social media coverage didn't affect the science or the scientists, who had spent the past decade conducting research on the subject.
I will maintain that Twitter/X/Bluesky are not part of the scientific process, nor should they be. These platforms do not encourage objective thought or reasoned arguments.
Maybe the problem is with our funding model. Necessarily the whole grant system is based on being able to argue a narrative as to why your scientific inquiry deserves money. Combine that with a system that includes incentives towards or away from social values, and scientists are necessarily activists.
And then that’s just to get money in your specific direction, getting money in your general direction requires more broad activism.
How so? It seems obvious that you can do science (that is: attempt to advance the understanding of how the natural world works) without being an activist for any cause.
The job of science is to discover facts and produce new knowledge from those facts. Activism is the marketing of an ideology. They couldn't be more opposite.
Oh I mean I don’t care if the teacher is for or against school shootings, I just find it interesting to have subject matter experts share their knowledge and give their opinions on things that impact their field. Some people just don’t take it well when those opinions don’t line up with their own on contentious topics.
How is it "interesting" for an elementary school teacher to be against school shootings? I'd bet I can find some carpenters who are against smashing thumbs with hammers, but why bother?
Are you calling it "Activism" when someone shares the opinion of 99.9% of the population, and spends 0 time advocating for that opinion?
Auto mechanic: Consumer advocacy, business regulation, labor issues, safety, etc.
Professor of Medieval history: Lots of political discourse makes claims about history or things like "the dark ages" that turn out to be mis-interpretations or false. Note that I have a friend in that field who often writes gentle corrections to false historical claims in online discourse.
When one political party is explicitly anti-science in its goals and aims and actions (RFK, global warming as a hoax, anti-vaxx, COVID as a hoax), Nature endorsing the person who is pro-science isn't political; it's existential. This is not "no reason". It's just not the reason you like because for some reason.
I bet they’ll follow the sports media and, my personal taste in sports media, has recently started to migrate (e.g., Defector editors and contributors).
In my experience, the change has been positive. I rarely see feuds on Bluesky and when they happen I find them especially embarrassing because it’s so unusual.
I stopped posting on Twitter around the acquisition but kept my account. When I do randomly check my timeline I’m genuinely disturbed by the disinformation and pseudo-science, especially in machine learning.
In general that's true, but it's because on Bluesky, blocking causes the blocked person's replies to be hidden from everyone who views the original post. It's much easier to block than to engage with a contradictory reply (especially if the reply is correct and you're wrong), so disagreement tends to result in a block.
This behavior is common enough that it creates a chilling effect for anyone who disagrees. Why take the time to craft a reply correcting the poster if it will likely be hidden from everyone? And so you end up with echo chambers.
The effect is quite stunning on some topics. For example: Quite a few people on Bluesky believe the Trump assassination attempt in Pennsylvania was staged[1], that the Charlie Kirk assassin's text messages are fake[2][3], and that the recent ICE shooter was a false flag.[4][5][6] Notice the amount of engagement these posts have. Thousands of likes, with little to no disagreement in the replies. The lack of feuding is what allows people to believe these falsehoods.
And the issue is bigger than it looks since blocking is public, so blocking gets you on lists of users to block so you'll be blocked by people you never interacted with for blocking/disagreeing with someone.
I don't know that I really want to interact with anyone who uses a block list like that, but it definitely would make echo chambers worse.
I've seen the same thing happen in smaller discussions among experts on Bluesky, but I didn't link to those because: 1. It's harder for non-experts in that field to judge whether my claims are true. 2. It might reveal my identity.
The incentive structure is the same as larger discussions. If anything, a smaller community makes it easier to create echo chambers, as you need to block fewer people before reaching epistemic closure.
What’s the problem using it as a switch statement if you care about typographic issues? I do this so I’d like to know if I missed something and this is a bad practice.
It's not an issue, but that's not where most of the power is and can also be confusing since if you use variables in the case statement, the way it watches does not behave like a simple switch.
