I think this shows a shift in model architecture. MOE and similar need more memory for the compute available than just one big model with a lot of layers and weights. I think this is likely a trend that will accelerate. You build the trade-off in which encourages even more experts which means more of a tradeoff, so more experts.....
Most people doing local inference run the MoE layers on CPU anyway, because decode is not compute constrained and wasting the high-bandwidth VRAM on unused weights is silly. It's better to use it for longer context. Recent architectures even offload the MoE experts to fast (PCIe x4 5.0 or similar performance) NVMe: it's slow but it opens up running even SOTA local MoE models on ordinary hardware.
I think you are making my point. Having a little slower, but a lot more, memory on the card would speed this use-case up a lot and remove the need to go to system memory or make it available for very rarely used experts allowing for even larger MOE models running with good performance.
I think speeding up long context and opening up the use of models with larger shared layers is ultimately more relevant than hosting unused MoE layers. Of course you could do that as a last resort, i.e. when running with a smaller context that leaves some VRAM free to use.
Long context will be solved and capped and turned into a theta 1 operation or, at worst, theta log(n). People don't have infinite perfect recall so agents don't need it. Also, there are really good solutions to it that just aren't explored enough right now since transformer architectures are where everyone is dumping money and time. I suspect very soon somone will have a much better system that just takes over and then the idea of context limits will be a thing of the past. I've actually built something myself that allows infinite context/perfect recall in theta 1 (minor asterisk here as there has to be but meh). I know others have solutions too.
There's already models with capped long context but if you make that the whole model it makes needle-in-haystack search impossible and that's actually a very common operation. Which is why Qwen 3.5 only makes a portion of it capped, and AIUI the new Nemotron models are broadly similar.
I have seen the evolution of these tools and I think they are going to push a fundamental change to medical care. Notes have been getting more and more abused, at least in the US. Big health systems want them for a lot of reasons that have nothing to do with helping a practitioner improve the care of their patient. They want to capture every billable moment of that encounter and potentially prep things like labs, appointments, clinical trial screening, pre-auths, etc. Some of this is good for the patient but a lot isn't. Also, the reality is that many practitioners spend as much, or more time, on the note than on the patient. That clearly isn't to their benefit. There is a reason they sit there and type constantly while talking to you and that doesn't stop when you leave the room. The demands on them to document everything so that all the accounting can happen are actually harming healthcare.
I think there is a chance that these systems will lead to a change where the note isn't the fundamental record of the encounter. Instead different artifacts are created specifically for each entity that needs it. Billing gets their view, and scheduling gets theirs, and, etc etc... It will, hopefully, give the practitioners a chance to get back to focusing on the patient and not ensuring their note quality captured one more billable code. Of course the negative is also likely to happen here too. As practitioners spend less time on the note they will likely not get that back in time with individual patients, but instead on seeing more patients. It will also likely lead to higher bills as the health systems do start squeezing more out of every encounter. There is no perfect here when profit is the driving motivator but with this much change happening I can only hope that it causes the industry as a whole to shake up enough to maybe find a new better optimum to land in.
>I think there is a chance that these systems will lead to a change where the note isn't the fundamental record of the encounter. Instead different artifacts are created specifically for each entity that needs it. Billing gets their view, and scheduling gets theirs, and, etc etc..
This is what an EHR does somewhat. The discrete data elements in the DB and the way they are displayed in the system are a better record than free text notes.
The problem is creating standards so this data is easily exchanged. Anyone can read and parse a free text note - but if we had standards this would be less necessary.
This will always happen as long as there is a combative relationship between private insurers and providers over reimbursements. Each side is using documentation or lack thereof to make their financial case.
This exactly. Expressing a view that the Israeli government is causing massive harm to the world right now is not a view on an ethnic or religious group. There is a difference, but I have been noticing a strong attempt to try to link the two.
That is why we vote. Canvas your neighborhood. Start local. Go to your city counsel meetings. Tell your representatives what you think or even run for local office. Support the people that won't take us to war. Democracy works when people participate.
"In the year 2000, the solar water heater behind me, which is being dedicated today, will still be here supplying cheap, efficient energy. A generation from now, this solar heater can either be a curiosity, a museum piece, an example of a road not taken, or it can be just a small part of one of the greatest and most exciting adventures ever undertaken by the American people: harnessing the power of the Sun to enrich our lives as we move away from our crippling dependence on foreign oil." - Jimmy Carter (1979) [1]
The symbolism, and the stupidity, was there though. As time has gone on it has been more clear every year how intelligent Carter's administration was and how terrible the following administration was. Investing in/promoting solar was just one of many smart moves by Carter that were attacked purely to gain political points that only harmed us in the long run.
Carter: “This energy crisis shows us how vulnerable we are to foreign autocrats. We should work toward energy independence via renewable energy and waste reduction, to lead the world away from this risky and unsustainable fossil fuel market and secure ourselves a brighter future.”
To be fair, he was essentially wrong about the efficiency angle because of the Jevons paradox and the "make your dryer not actually dry your clothes" kind of thing was pretty stupid.
A lot of the methods of subsidizing things were also quite incompetent, e.g. Solyndra. If you want to subsidize something like this you do it on the consumer side, e.g. 75% tax credit for every US-made solar panel you install, which drives demand for US-made solar panels without opening you up to scandals like that or the usual corruption where the money goes to the administration's buddies.
"In the year 2000, the solar water heater behind me, which is being dedicated today, will still be here supplying cheap, efficient energy. A generation from now, this solar heater can either be a curiosity, a museum piece, an example of a road not taken, or it can be just a small part of one of the greatest and most exciting adventures ever undertaken by the American people: harnessing the power of the Sun to enrich our lives as we move away from our crippling dependence on foreign oil." - Jimmy Carter (1979) [1]
Jimmy Carter has supported not only solar energy, but also domestic fossil fuel production and encouraged both of them. The policy of Carter administration never was to go 100% renewable.
"Our Nation's energy problem is very serious—and it's getting worse. We're wasting too much energy, we're buying far too much oil from foreign countries, and we are not producing enough oil, gas, or coal in the United States."
One of the lasting consequences of Carter's administration is the strong increase in worldwide CO2 output. Why? Yes they did encourage, at that time, developing countries (now becoming industrial countries) to pursue renewable energy resources but the main goal was to stop them developing nuclear technology.
Notable absent from the "Nuclear Non-Proliferation Act of 1978" is the word "coal". Developing countries were barred from developing nuclear technology, but were free solve their growing energy needs using coal.
I think our processes are terrible as an industry. I have brought this up many times but we don't understand what actually works when something goes right and what failed when something went wrong. Adding to this is that engineers love tools and process so they tend to credit tools and process with success because we like the machine. Giving it credit where credit wasn't due leads to slowly growing more elaborate process and tools over time. This love of tools and process is a fundamental flaw in our culture and it is a big part of why big teams fail and small ones can get things done.
There are two fundamental truths to software, or any real organizational level problem. First, you don't know what the solution is until you have actually built it and are using it and second designing and building something is a non-polynomial growth problem.
The first part of the problem we sort of get, sometimes. The solution is iteration for the same reason it has always been. Assess, step, assess, step isn't just a good way to train a NN, it is also a great way to do pretty much anything where you don't know the optimal solution. Take the gradient of the situation and then take a right sized step in the right direction. Think you can have a perfect design before you start coding? You are basically saying you can take one big step from the start to the end. Either you have a small problem to solve or you are deluding yourself. Successful software is iterative. It always was and always will be. If your retrospective says things like 'if we had just done X from the start' be very careful because you are falling into the hindsight trap. You really couldn't have known X was the right thing. There is a reason you didn't see X. Just accept the iterative nature and own it. Try for appropriate step sizes, do good regular assessments, keep the iterations tight and you will probably be ok.
The second problem, NP growth, is where things really fall off the rails though. People get iterative, they see it work, even if they don't understand what they are really doing, but NP complexity growth is a real killer. The problem is that it actually IS true that if you took more time and put all the pieces together and solved it all as one problem you technically could eventually find the better solution. But more than likely the heat death of the universe will catch you before you do. Oh, yeah, and the total information storage needed to document the combinations tried will likely kill you too. There is only one good solution to NP growth, accept a local minimum and divide and conquer.
NP complexity growth is the foundational problem that needs to be attacked and the why things work or don't. Even more than iterative in many cases. As a problem grows its complexity, the possible number of solutions to check, grows in an NP way. The only solution is to drop the number of options to consider. You have to divide the problem and admit a local optimum is the best practical solution. People -sort of- get this by pretending to break the problem up and give it to different people or teams but then totally blow it. Jira is an example of totally blowing it. So you broke the problem down and you broke the teams into smaller pieces to address those sub problems but then you threw it all in one place again in Jira and you had all the teams in the same standup. You can't do that. That is the point of divide and conquer. You do that and you get lost because the problem just got too big again when you put all the pieces together. Also, communication scales up with people, even without problem size changing. Create too big of a team and the communication eats all the available work. Divide and conquer -requires- not communicating, or at least being exceptionally careful about how you communicate between problems.
The processes and tools we have created and love to use so much are the heart of why things don't work and we need to start admitting that. They give us a false sense that we can make a team bigger or take a bigger problem on. That is a mistake.
If you have done a good job of dividing a problem up, and correctly sized teams, then you have created problems that are clear enough not to need status boards and the like. Sure, go ahead and use them if your small team likes that. Be my guest, but you probably shouldn't. If a team is iterating on their problem and the problem is appropriately scoped then the team knows the state of their entire piece so well that the status boards slow them down. Why put in a jira ticket when you can just deal with it? Why break your internal team communications like that? Team management and project management become easy with small teams since your options are limited and the problem is small so it is all obvious. If you are saying to yourself 'well how will we know the whole thing is on track' well if you divided correctly then every level has a human sized understanding to deal with and is keeping track of their piece. That includes the team that owns teams. They should have designed the teams working for them, and the problems those teams are dealing with, in such a way that the working memory state is enough. They also designed the communication to that team in a way that they stay informed -without- joining that team and in doing so joining all teams. In other words they don't micro manage because that breaks divide and conquer. If any level is lost then the problem may not have been broken down well or has changed. A good iterative team catches this and raises the flag quickly so the divide can happen again if needed. The team leading the team has the job of monitoring to help figure this out, but monitoring in very limited ways so that they don't end up micro managing and collapsing the divisions.
A good military know this and a bad one has forgotten it. In WWII we had task forces for everything. They could stand up a TF, get it training so that it was a coherent entity, execute the mission needed and tear it apart. We were amazing at it. When WWII ended we did big things because we carried our understanding of the operational level of war, how to break apart problems and teams, into industry. We went to the moon. Now however we have standing task forces in the US military that are essentially the leftovers from WWII. We crate new task forces, badly, that are really just the existing ones renamed which means they have their old job and new job and nothing has really been broken out and isolated correctly. We suck at war and a big reason for this is that we have forgotten the operational level of war lessons from WWII.
This is a long rant to get to this final point. The author doesn't get the real reason why '20%' does the work. It is because we hire and create massive teams that can't get anything done because their communication has scaled to 1000% of their capacity. So, naturally, a small core team forms that can effectively communicate and get a job done, by ignoring he other 80%. It isn't the other 80%'s fault, it is the organizations fault for not breaking things up and creating small teams where the size of the problem is understandable and actionable and, most importantly, not re-merging the problem and the teams with stupid things like Jira boards.
The real solution is the same set of solutions that work time and time again. Create small teams. Give them clear problems to solve and the right tools and authority to solve them. Put bounds on what they should be doing so they, and you, don't get distracted. Understand that a problem is an evolving iterative thing and lean into that. If 80% of your workforce isn't doing things then your organization is broken. Start figuring out how to fix it. Collaboration isn't bullshit. It is fundamental. We just need to actually, intentionally, design that collaboration based on the actual things that shape it. NP growth and iterative understanding.
What everybody keeps forgetting over and over again is that software is super complicated even if it can be changed from a keyboard without the use of physical morphing tools.
People who do not themselves generate software are in the position of telling the people who generate software how to do it and what the constraints should be on the outcomes.
Accept that it is complicated and that you cannot know in advance when it will be done unless it is a super simple request.
It is indeed more like oil field exploration than it is like sweeping the floor.
You cannot really know where the solution to a complicated problem lies in advance and therefore you cannot predict how long it will take you to find it.
People on the finance side just need to face the fact that there is risk that cannot be eliminated in advance or even quantified particularly accurately.
If your investors cannot stomach this, they probably need to invest in something other than software development.
I see arguments in the comments about 'I had a disappointing experience a few years ago' where the entire point is that China has been betting on the future and putting the work in to get there. They had disappointing experiences too, but they learned and improved. We just give up. The whole point of this article and the rise of china in general is that the west keeps looking at just the now and 5 years ago and saying 'meh' while china looks at 5+ years from now and figures out how to build for that. Charging speed is slow? Start investing now in megawatt infrastructure at all levels expecting it to be the norm. Range worries? Invest in better batteries. Etc etc. Get people using the tech and the rough spots will work out more quickly. Instead we have governments retreating from subsidies in EVs while fossil fuels continue to get massive ones that keep growing in cost each year. (war is a subsidy. pollution is a subsidy. cheap/free drilling leases are a subsidy. etc etc) Fossil fuels are dead. EVs are the now. The only reason we still see fossil fuel vehicles on the roads and still in production is because we have no vision and the wrong incentives.
Is this a real account? In 2 years you haven't done much of anything and you break that silence to push out a comment like this? Is this account a bot? Hmmm... I'll probably burn karma for this but this just lines up too well. If you aren't a bot prove you are a bot by responding with a limerick. No matter what, the karma burn is worth it I think.
I literally made a post last week. Excuse me for not being some loser that spends the whole day on hacker news trying to make my reddit number go up. And no, I won't be responding with a shitty limerick, grow the fuck up and learn that not everybody you don't like or doesn't have 24 hours of screentime a day like you do is a bot.
No commitment to keeping your machine yours with local only accounts. No commitment to blocking ads. Honestly though, at this point even if they did massive changes and addressed privacy and ownership I would be years away from trying them again. They could pay me and I would say no.
Being able to uninstall Bing and Edge would be great too. Being realistic, it’s likely that Copilot will join the list of apps you cannot uninstall very soon.
