Something I've learnt the hard way.
While there is merit to be able to do things without model assistance, this is a tale as old as time: purists who resist model usage will in the end be the unintentional enforcers of a culture of: everybody uses AI but the ones who come on top are people who pretend otherwise/who can best hide the fact that they are using them
You're probably aware of things like hyping up your resume, hyping up your stories and the existence of people on the other side who know of this dog and pony show and continue to play along?
What you're going through in my opinion is the AI age analogue of that: everyone probably uses them but the people who come on top are the people who are able to pretend they don't
There are many papers that use a recurrence across sub-sequences and attention within sub-sequences. Google did this with Infini-Attention and one of the variants from the Titans paper. However, I think the earliest example of this is Transformer-XL.
There are some interesting connections between them. If you remove the softmax from the attention formula, you end up with linear attention, which has a recurrent form.
I haven't read it, but the Mamba 2 paper claims to establish a stronger connection.
Given how o3 cracked the arc bench and I'm probably sounding like a broken record, this isn't as farfetched as some of you may think it is. ML models will very likely continue to scale regardless of how many bets are placed against it. I'm not sure why a lot of people aren't concerned about arc bench being cracked so fast. Our grand delusions of specialness has been shown to just that, delusions
"Humanity is a just a small step in the giant staircase of intelligence" - Geoffrey Hinton
I have no clue if AGI will look anything like today's LLMs but I don't think the information we have about o3 so far suggests that it's particularly earth shaking or even a significant step towards AGI.
From the ARC announcement: "a large ensemble of low-compute Kaggle solutions can now score 81% on the private eval." If I understand this correctly, o3's performance is not a grand leap beyond the capabilities of many times cheaper models with similarly privileged information. The ARC news seems more likely to be evidence that the benchmark needs tweaking than proof that scaling works (although OpenAI's marketing team would like us very much to interpret it as the latter).
There has also been a bit of imprecision and hand waving around other benchmarks that bolsters my skepticism. For instance the Codeforces benchmark results were touted with no meaningful description of the methodology and what little we do know suggests (to me, at least) that comparing o3's elo to that of a human is an apples to oranges comparison: https://codeforces.com/blog/entry/137539
In my (possibly flawed) interpretation: o3's scores appear to be an achievement because they were attained by a single model, but the benchmark itself needs refinement before it can claim to be a measure of AGI like it set out to be, as one can bruteforce their way to similar results.
> you're saying you can't understand why you should be happy to be here?
I think you miss the aspect of how insignificant existence is locally.
Sure the odds are astronomical but you weren't there to experience that measure in it's entirety i.e the the billions of years for you to experience the specialness of the blip. Also compare that with billions of people current existing at the same time as you who are also the product of this astronomial odds. The awe of the statement of the specialness of existence quickly fades away when you take the former statement into consideration
One could take this in the opposite way, we're so special that we are barely get to live long enough to experience reality for what it is and have to make do with such a tiny drop. The unfairness of it is misery inducing. We are so special that we get to appreciate this specialness only if we're lucky enough to be born in a first world country and to decent parents and born healthy. Aside from that we have spent a very significant time sleeping, pooping, dealing with BS, dealing with things out of our control etc etc
Existing is truly miserable if you aren't living in a first world country.
Man your statement is just hollow. The astronomical odds of existence is nothing celebrate by itself just as hope by itself is useless
Is it not true that The Arc test is designed to be one where the rules are dynamic? i.e every one of the tests are different from each other in an absolute sense. Learning about one tells you nothing of substance about the other unless of course you/the model is capable of meta-learning
Finetuning has been looked down upon because all it does is rearrange weight to learn style of the finetuning dataset. It does not teach the model anything which is in contrast to the hopes behind finetuning
If a model was able to ace the arc-test just by the merit of being finetuned, does it not imply there is something of absolute substance here? i.e the model is capable of meta-learning and all it needs to adapt to a new-task is a bit of finetuning which again I emphasize is the loweest tier in the ranks of types of training models
I think "manifolds" in AI are not the same as actual smooth manifolds. For starters I would not expect them to have locally the same dimension across the whole dataset.
What I meant is that I can imagine cases where some part of the dataset may look like R2 and then colapse to have a spike that looks like R1, so it is not a standard manifold where all of it has the same dimension.
Appart from that, these "manifolds" have noise, so that is another difference with the standard manifolds.
I think there needs to be some feedback that the goal is sustainable, one tooth a day to an adhd mind is just as bad as not doing anything as it's still a very tall staircase to climb. Too granular a step
There is a reward at the end. That is an important requirement for this method.
It’s easy to try this out in a little more structured way and with assistance. Tiny habits [1] has a free five day program where you can try it and seek assistance from a person trained in this method. I tried it and it didn’t really stick for me, but I wouldn’t say that it was a total waste of my time. I could relate to the advice and I plan to read the book sometime.
It really feels like a scaling problem than anything else. It big companies go to shit as they do most of the time it's because tackling scale is a extremely difficult problem
They have to scale, and they don't see it as a problem.
Like the author says, at a certain point, you've saturated the market. Everyone who wanted M2 Macbook Pro has one, but still, the institutional shareholders come knocking, demanding more money so that retirees can move into that villa on the golf course in Florida and consume until the dementia sets in.
So what do you do to keep the shareholder returns coming? You expand into new markets, or create new markets, or create new products. Thus, you grow in scale.
Again, to them, this is not a problem. Most of the people involved in the actual workings of this system - executives, fund managers, financiers, etc. - are getting more money out of it than they could ever hope to spend in a lifetime. There is no downside the system produces that their wealth cannot overcome. The rest of us... well... we're annuities.
> demanding more money so that retirees can move into that villa on the golf course in Florida and consume until the dementia sets in.
There are also the taxpayers that need the government pension funds to make good on investment assumptions, lest taxes have to go up.
Edit:
Politician A: I will remove all deferred compensation schemes, but have to increase taxes to pay the employees with money today
Politician B: I will pay the employees with deferred compensation, and keep taxes lower than politician A, and let future taxpayers deal with any problems of underfunding and corruption
And politicians who keep making promises about a solvent pension fund because it's not fun for them if it's not solvent, and unions/retirees are much better at voting than the average person.
You're probably aware of things like hyping up your resume, hyping up your stories and the existence of people on the other side who know of this dog and pony show and continue to play along? What you're going through in my opinion is the AI age analogue of that: everyone probably uses them but the people who come on top are the people who are able to pretend they don't