> Whenever you try to force the real world to do something that can be counted, unintended consequences abound. That’s the subject of two new books about data and statistics: “Counting: How We Use Numbers to Decide What Matters”, by Deborah Stone, which warns of the risks of relying too heavily on numbers, and “The Data Detective”, by Tim Harford, which shows ways of avoiding the pitfalls of a world driven by data.
Data is a powerful feedback mechanism that can enable system gamification; it can also expose it. The evil is extracting unearned value from a system through gamification not the tools employed to do so. I’m looking forward to reading both books.
> woodworkers take pride in the finished product, and seeing that finish product being used. They are not all hot on the tools they use, and nor do they value the art of planing for its own sake.
Woodworkers are hot on the materials that go into their finished product and, for better or for worse, our language choice brings along a runtime and library that becomes the foundation of our digital artifacts.
Our toolchain matters and mastering more than one is non-trivial. Material world metaphors don’t map well to the digital world. I’m not a fan of Uncle Bob Martin and the tweet about Clojure in 2021 made me giggle but a mediocre proposal doesn’t take away from the importance of the underlying problem; our language ecosystem mastery defines the problem spaces we inhabit.
The paper is pure science. I’d argue that the Life Sciences are entering a Scientific Revolution 2.0. We are on the cusp of learning how many living systems work. The scope is broader than genetic engineering alone, IMO. This revolution is shaping up to be multidisciplinary; computation will be a critical aspect.
Agreed and I think the manipulation of gene and rna is revolutionary. It might be just steps and tools. But if not control, do we still have human is an issue.
The Smithsonian article is based on the paper The Four Black Deaths [1] by Monica H. Green:
> The Mongols, whose empire emerged in 1206, unwittingly moved plague through Central Eurasia in the thirteenth, not the fourteenth, century. Grain shipments that the Mongols brought with them to several sieges, including the siege of Baghdad, were the most likely mechanism of transmission. The fourteenth century plague outbreaks represent local spillover events out of the new plague reservoirs seeded by the military campaigns of the thirteenth century.
The article focuses on server-side Node, rather than its role as a CLI tool. Compared to V8, Ruby, Python, and PHP are not performant. Node’s libuv network library was highly concurrent and Node built single-core concurrency into the runtime and libraries; multi-core concurrency, however, is not best-of-class.
IMO, Ryan Dahl made two fundamental mistakes in both Node and Deno that make them uncompetitive for simple server-side 12-Factor Apps: 1. No web Gateway Interface like WSGI and Rack, and 2. No DBMS API/SPI like JDBC. Express was supposed to be a Microframework like Sinatra and Flask but developers have to spend most of their time defining a robust web server. Since the http server and app code are tightly integrated, 12-Factor Apps or JSON API Microservices are more complicated than they need to be in Node. The lack of a JDBC-like API/SPI hurts many languages included Node and Rust.
The GIL languages are not performant nor concurrent but they have good httpd/dbms interfaces. All the popular server-side language/runtimes are flawed along some fundamental dimension. Server-side REST shouldn’t be this complicated.
It uses http, which is much more standardized, used and accepted than Rack or WSGI. I can chain together http servers, use graphql http redirectors, etc. With WSGI and Rack you’d need a fronting web server to terminate and translate.
> No DBMS API/SPI like JDBC.
Covered this in another comment.
> Express was supposed to be a Microframework like Sinatra and Flask but developers have to spend most of their time defining a robust web server.
Doesn’t this mean it is a microframework? As opposed to something like Django that has everything included?
> Since the http server and app code are tightly integrated, 12-Factor Apps or JSON API Microservices are more complicated than they need to be in Node.
If you are serving up an express app directly with no fronting load balancer then perhaps they are tightly integrated. Otherwise they are not. Express does not and should not handle load balancing.
Node.js wasn't created in a vacuum. Its http lib, and express.js which extends it (and Sencha connect before that) implements part of the CommonJs API created by earlier SSJS frameworks such as Narwhal, Helma, etc. and was inspired by Ruby's Sinatra.
In fact, a common portable API and standardized language that isn't going away anytime soon (JavaScript) is what draw me to Node.js, and I still find it's an excellent framework for lightweight web servers and b4f approaches. TypeScript and Deno, not so much.
Edit: also, while Node.js had its share of quirks in early versions (especially around the Streams 1/2/3 API), its core API is super-stable (and it better be with some 100s of thousands of packages making use of it out there).
It was my understanding that Node never implemented JGSI [1]. I've never compared the two APIs so you may be correct. Regardless of the path taken, the middleware should apply to a separate web server module that is functional by default. The tight coupling that emerged was too hard to master and the defaults are too attack prone to inspire quick public deploys. It resulted in copy-and-paste server-code between projects rather than reusable server modules.
This may be a property of all embedded network libraries but I think my observation still holds; server-side Node did not outpace the alternatives despite its superior performance/concurrency relative to the dominant GIL platforms.
EDIT: AWS Lambda (i.e. Function-as-a-Service) can be considered an extremely simplified service gateway interface though it is rarely thought of in those terms.
> middleware should apply to a separate web server module that is functional by default. The tight coupling that emerged was too hard to master and the defaults are too attack prone to inspire quick public deploys. It resulted in copy-and-paste server-code between projects rather than reusable server modules. ... did not outpace ... GIL ...
I honestly have no idea what you're talking about. The beauty in Node.js' http lib is that you can write middlewares against the core API and run the same code under express.js with additional middlewares for sessions, routing, etc. Python, whatever its strengths, certainly isn't used for web apps more than Node.js, let alone is dominant. Re modules and reuse, I guess if one side is complaining about too much modularity (leftpad) while the other doesn't see enough of it, Node.js got it just right ;)
Edit: CommonJs directly references JSGI [1], and if you compare it to Node.js' http core or express.js' API, you can see the correspondence quite obviously, can't you?
Modern PHP frameworks (Workerman, Swoole, Comet) rank as top performers of Techempower Benchmarks. Much higher than Python / Ruby / Node.js and head-to-head with the Golang based project performance.
> The lack of a JDBC-like API/SPI hurts many languages included Node and Rust.
I always wondered what is a use case for Rust service that talks to a (non-local)database. First people pay borrow checker tax, get a chance to have an amazingly responsive system in return and then blow it on the round trips to a database. What am I missing?
Users who use Rust in this way report that these services tend to be extremely robust and have orders of magnitude less resource usage, which in a world of cloud computing translates directly into an improvement on the bottom line.
Because otherwise you end up with thousands of incomplete and incompatible implementations all fighting to be the best or running out of steam after a month or two. Node is a hellscape from a db perspective
I'd phrase it more like "ORMs are as popular as they always were, most DBs today are SQL DBs, same as always, and the only recent change in the last 10 years is people have stopped speculating that NoSQL is going to replace SQL DBs, and accepted they will be, at most, specialised tools for niche uses.".
"Most DBs" by what metric? If you count them as products, you're probably correct. But I seriously doubt that is true when adjusted for usage - say, number of requests per second, globally, across all deployed databases worldwide.
(in fact, I suspect that SQLite would win that contest handily.)
SQLite is in every android phone, it wins by a mile. I'd even go as far to wager that MSSQL instances alone outnumber Mongodb or Dynamo by a good margin.
I think that statement should be strongly disputed. Most businesses still have a SQL database at the core of their stack. NoSQL is still definitely the exception rather than the rule.
This can generally fall under the category of “Generate Test Data in SQL” with both generic techniques like Recursive CTEs, as in the OP, and SQL dialect or tool specific options. Search is your friend. The OP also provides a table lookup technique for readable names but doesn’t address other data types such as timestamps or other functions such as MD5(). Other data distribution techniques other than random may also be needed. This post scratches the surface of testing with generated data.
Indeed, the age group for cerebral venous sinus thrombosis (CVST) is very specific [1], mostly women < 55 (menopause?):
> The Committee’s experts looked in extreme detail at records of DIC and CVST reported from Member States, 9 of which resulted in death. Most of these occurred in people under 55 and the majority were women. Because these events are rare, and COVID-19 itself often causes blood clotting disorders in patients, it is difficult to estimate a background rate for these events in people who have not had the vaccine. However, based on pre-COVID figures it was calculated that less than 1 reported case of DIC might have been expected by 16 March among people under 50 within 14 days of receiving the vaccine, whereas 5 cases had been reported. Similarly, on average 1.35 cases of CVST might have been expected among this age group whereas by the same cut-off date there had been 12. A similar imbalance was not visible in the older population given the vaccine.
This seems to be yet another case where the news is incapable of keeping up with timely technical information. No mistakes were made, the pause could have gone either way and we should generally be more understanding of difficult but timely decisions made in the face of uncertainty.
At the heart of progressive thought is concern over the twin evils of oppression and exploitation. O'Reilly’s piece focuses on the latter but relies on the assumption that profit==exploitation. I’m not convinced it is that simple.
I wish I could have had a chat with Milton about the Friedman doctrine [1]. I want to give him the benefit of the doubt. From my perspective, shareholder value is a relatively easy measure that acts as a proxy for the true goal of a corporation: creating sustainable customer value.
Within the context of his original article, Friedman was arguing that customers, employees, and shareholders could choose to give to charity, for instance, but it is wrong for a corporation to do it on their behalf. The concept is not intuitive but it is far from promoting selfishness.
> as Theranos demonstrated so vividly, it is harder to sustain a hype balloon in a scientific enterprise than in many of the markets where Silicon Valley has prospered.
This narrative continues to frustrate me. Eleanor Roosevelt famously said:
> Great minds discuss ideas; average minds discuss events; small minds discuss people.
The big idea behind Theranos was that combining multiple microfluidic tests could produce a lab-on-a-chip. Theranos failed to combine more than a handful of microfluidics together and scrambled to find alternatives to bridge the gap.
Ultimately they were foiled by brain-dead regulatory tests that specified blood draw volumes of blood. This regulatory test has to change to accommodate fingerpick volumes of blood; a key feature of microfluidic tests.
Our collective small minds couldn’t focus on anything but the people involved. The core idea is still an outstanding hard problem.
> Ultimately they were foiled by brain-dead regulatory tests that specified blood draw volumes of blood. This regulatory test has to change to accommodate fingerpick volumes of blood; a key feature of microfluidic tests.
As I understand it, for many of the tests Theranos was aiming to do there's no real way to achieve it with the tiny volumes of blood they were trying to use: the concentrations of the substance they were testing for were so low the statistical variation of the amount in any given small volume of blood would invalidate the test, even if you could accurately count it.
> for many of the tests Theranos was aiming to do there's no real way to achieve it with the tiny volumes of blood they were trying to use
This is exactly the right question to ask. Right from the start we should have had a scorecard of the common lab tests doctors request (under 50 I think). For each test you should ask “is it possible with microfluidics?” and then “has it been combined on a single strip/card yet?”
I’m not sure if any concentrations are too low for microfluidics rather than the dilution required to meet the volumes specified in the standardized tests. Are fingerpick volumes too small or are the regulatory tests outdated?
The same issue applies to self administered rapid antigen tests during the pandemic. Regulators focused on the “gold standard" PCR tests and ignored the benefit of cheap and easy unamplified tests that can be used in a daily regiment.
They focused on the people because they were either scammers or well-known people with no domain knowledge.
The basic idea was very straightforward. I can come up with lots of great ideas if I ignore whether there's any reasonable path to a solution. Hey! Let's have a Star Trek medical tricorder implemented in a watch. Let me know when you're done implementing it. I'll be at the beach.
> They focused on the people because they were either scammers
The scamming came after the failure of the core premise of the startup: microfluidic lab-on-a-chip. My point was not to excuse the scamming, it was to switch focus back to the core ideas and not the people sporting black turtlenecks.
I suspect that if the original strategy worked the story would be different. Transparently reporting technical/business failure is an admirable trait but it is a different thing than innovation. Journalists and analysts should have been reporting on the scorecard (strategy/tactics/execution). When the company did a technical pivot why was this not enough to investigate the underlying cause? I'm a potential customer that wants this solution; I feel like I've been cheated by all involved.
QUESTION: if someone else succeeds with a microfluidic lab-on-a-chip can the regulators validate its efficacy?
Great data, mind boggling interpretation. This is exactly what I’d expect to see in Greenland’s subglacial sediment. It reinforces, not rewrites, the paleoclimate history during the 41 kyr to 100 kyr glacial cycle transition.
> Whenever you try to force the real world to do something that can be counted, unintended consequences abound. That’s the subject of two new books about data and statistics: “Counting: How We Use Numbers to Decide What Matters”, by Deborah Stone, which warns of the risks of relying too heavily on numbers, and “The Data Detective”, by Tim Harford, which shows ways of avoiding the pitfalls of a world driven by data.
Data is a powerful feedback mechanism that can enable system gamification; it can also expose it. The evil is extracting unearned value from a system through gamification not the tools employed to do so. I’m looking forward to reading both books.
[1] https://archive.is/ynOm2