Daniel Dennet in „Consciousness Explained” argues that consciousness is an emergent phenomenon, and when we look at its individual components, it is like seeing an illusionist’s trick.
We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)
The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.
- How many individual components do you need for it to emerge?
Most people would be okay with saying that individual cells are not conscious, maybe even that tree are not conscious even thought they are made of many cells. Neurons seem to be the determinant factor in deciding whether something could have a consciousness, but again how many do you need? Does growing 1B neurons count as a brain if they are not organized?
>- How many individual components do you need for it to emerge?
If it is about the relationship between components, then I would imagine just two. Then it is a matter of scale.
This seems to be anathema to many people. I'm not sure why but the notion of something having a tiny bit of consciousness that is imperceptible seems to be unacceptable. There are so many things that we cannot comprehend at small scales. Nobody really has a handle on how large a Planck length is.
For some reason it comforts people to think there is a threshold at which it all switches on, but for what reason would there be a threshold?
Neat. Not sure if your site is a gold mine inside a rabbit hole, or a rabbit hole inside a gold mine, but I really dig both the aesthetic and the content.
I like this essay, and have very often referred to it when someone talks about AGI.
There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds.
It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.
Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.
So, less God, more WAU from SOMA or the Ocean from Solaris.
For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.
The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).
An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.
Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.
If you're gonna look at it that way, then the halting problem is just a dressed up computer science version of the question "can God make a rock so heavy even he can't lift it?" Could an infinitely smart AGI come up with the answer to the unanswerable question?
It doesn't need to be infinitely smart to do a better job than the worst of humanity's blunders.
But isn't on the other hand the current AGI problem posed similar to the question "Are you not afraid that genetic engineers grow babies with bigger and bigger brains?"
We don't know if that won't break down somewhere. Looking at different examples it probably will. Scaling things infinitely is a pure math only concept it seems.
Put another way, you do not deny a proof by inference because it "leads to large numbers".
We don't need to speculate, we can see many, many examples today of more and less intelligent species, and also what happens to the less intelligent ones, even taking humans out of the equation.
I'd argue, even a machine intelligence that merely managed to be mildly smarter than us would be a threat. AGI merely has the potential to be infinitely smarter than us, but that's somewhat irrelevant given we might not even be smart enough to realize how much smarter they are (a cat will likely not appreciate the difference in intelligence between a dog, an elephant, or a dolphin, despite all those animals being generally smarter).
Infinities aren't a physical reality. Resource are always limited, physics is limited at the Planck scale. You can only do so much compute in a finite volume of space, and there will only be so much energy available.
As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.
They are more characterized by how they grow and when they stop than by their "physical reality". Proof of this is in that different infinities exist - characterized precisely by how fast they grow, i.e., one "infinity" is "larger" than the other.
My point is, getting hung up on "infinity" as being unrealistic is not the point. It is the tool with which to understand how thing behave. The same as any calculus problem - you take the limit to the infinity to understand how the function behaves.
> We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.
I've only heard the God narrative from skeptics as in look at those idiots thinking they are building God. I think most people who believe in AGI arriving see it more as something like a chess computer but as well as beating us at chess it'll do other thinking too. A souped up chess computer isn't God.
Would you care to share any links? Specifically any talk of an AI-God "judging". If your entire perception is based on what you think you know about the Roko incident, may I suggest that you are ill-informed about it? It wasn't something the "LW crowd" endorsed or believed. Looks like I left some links to help someone else become informed several years ago: https://news.ycombinator.com/item?id=18982933
I can see how talk of a paperclip maximizer, or any sort of AGI that can actually deliver on e.g. Drexler's nanotech or otherwise act in very powerful ways like, to quote, "colonizing the galaxies", pattern-matches to something roughly omnipotent by most of our standards. I can see how it roughly pattern-matches to something like "immortal", but this seems like the least of feats, much of our complex machinery with maintenance is effectively immortal already. Overall I don't see how this connects to what you originally wrote: "There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds." It is not omniscient, it only knows enough to develop the tech to enable paperclip maximization. It is not judging, either: "The AI does not hate you, nor does it love you, and you are made of atoms it can use for something else."
I should also note that the paperclip maximizer is not something the LW crowd believes should exist. Its primary function as an idea is to illustrate the orthogonality thesis: that goals and intelligence aren't dependent on each other. Its secondary function is to illustrate instrumental convergence.
> In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.
Oh boy couldn't agree more. The whole Basilisk drama really caused me to rethink the idea that a group of Rationalists were in practice, were in fact particularly rational. CF The Atheists vs someone that happens to be atheist. To their credit, they also acknowledged this somewhat and hence the CFAR crowd.
The word "rehash" also resonates. Like Google has a tendency to use PHD's to reinvent everything over and make up new words and terms for the same concepts.
Having spent a little too much time in the early 2010's at Bay Area LW meetups, the more fantastical LARP of fanfic, and religious mythology often felt a bit at odds. There was the aspect of a charismatic, autodidact leader obsessed with a certain J.K. Rowling IP and the kinky stuff..Don't get me wrong, I still have fond memories of this time overall :)
From my perspective, a core issue seemed to be no-one seemed to particularly motivated in defining what it meant to be rational, aside from some loose segmentation around instrumental vs epistemic rationality. (ie practice vs theory). And because of this it almost had a faith vibe to the scene. Like "trust me bro" this is "super high brow nerd stuff" and on your third helping of "The Sequences" all these formulas and shiny new words will all make sense what and it will be clear why we’re doing these meetups. (It totally wasn't anything to do with mental masturbation and high-iq crowd bonding and feeling good ;)
When I was into Christian apologetics (C.S. Lewis etc) as in my first year of CS & philosophy of science in college, there was a similar thing.. after a year of seeking out the scientific, and logical explanations for all the religious dogma I was indoctrinated into growing up. In the end, it pretty much reduced to "just have faith". This was after exhausting the "well you're not an expert on Christian theology, so you cant have a solid argument around the nature and existence of God because you need to study more" counter. This despite reading and studying the Bible at length.
For example, right now. Is it rational to be typing this up on HN, when I have other more important goals to do? OTOH reminiscing on the past and connecting with a single serving online friend OP, gives a bit of a dopamine hit. And maybe sharing resonates with others and increases happiness in the world? (or not if anyone still LW reads this and maybe feels a different type of way)
So that’s community right and good (in the sense it's aligning with goals)? But then its driven by emotions, so that kinda is not supposed to be rational. Is it rational to observe one's own mental states and take action? Turtles all the way down!
I remember "Baba is Eval" (https://fi-le.net/baba/), released 11 months ago, back when Claude Opus 4 was the strongest model. Back then, I was surprised how poor was it even at the first level.
I am happy to see an another approach - and indeed, with much stronger results.
The whole title is a buzzword cluster, until proven otherwise.
Which tasks, in particular, does it do better? Not as in "it could do them better", but actually there are benchmarks. If they are, they are buried beneath marketing; if not - well, we have our answer.
What is "thinks like nature"? Spin systems, are no more (or less) nature than transistors.
That said, I am all for exploring various systems for computation and simulation - I think there is a lot to discover.
Yeah, I mean it's obviously meant to be a marketing pitch but it's not a very good one.
> The hardest computational problems are not waiting for faster chips – they are waiting for machines that compute in a fundamentally different way.
Surely they don't actually believe that, right? Like you say the benefits must be limited to specific shapes of problems (not all of "the hardest" ones), and the whole history of computing is about how faster chips is an excellent answer to difficult computational problems.
> and the whole history of computing is about how faster chips is an excellent answer to difficult computational problems.
I don't really disagree, and I am definitely not taking their marketing pitch seriously. Yet, you could look at the same computation history and interpret it as an economically constrained hill-climbing around an idea that was simple enough to work reliably (von Neumann architecture) and that worked and scaled so well that we were rarely forced or desperate enough to move conceptually far away from it.
Sufficiently general digital computers can simulate other computational models, so I think 'faster' is ultimately the end game, but for some classes of computation, as you also noted, we may need to go for analog hardware, (maybe) quantum devices, optical interconnects, and so on.
Bret Victor has a talk about this, more or less: [0]
That's interesting, thanks. I only read the abstract so far but was immediately reminded of this recent HN submission[1] and the whole thing that certain ideas go together, and so they are adopted together, but the resulting bundle of ideas might be poorly suited to certain problems.
I love how the game SOMA deals with taking (or not taking) someone's, or something's, life. It is never an easy moment. And with no commentary (with some haunting exceptions, https://youtu.be/pmC6naegRgo?t=161), no external reward or punishment, you need to carry the burden of your own decisions.
Some people says they will be judged (only) by God and history. In SOMA, there is neither.
reply