> [Arrow] adoption will take time, and most people are probably more comfortable seeing NumPy arrays. Therefore a Vaex version 4 a DataFrame can hold both NumPy arrays and Apache Arrow arrays to make the transition period easier.
There seems to be agreement that Apache Arrow is the future of dataframes across ML ecosystems. I didn't realize this transition impacted NumPy arrays in addition to Pandas dataframes in Python.
Wow, I'd like to hear more about the subsea cable [1] technology used. In terms of economics and geopolitics, this sounds like a win-win scenario. Perhaps necessity will force the eastern Mediterranean to re-emerge as an economic powerhouse. Southern Italy, Croatia, Turkey, and Egypt should pay attention.
> Perhaps necessity will force the eastern Mediterranean to re-emerge as an economic powerhouse
As a half Croat: sorry, won't happen. Croatia's economy is tourism dominated and will stay that way. There is a bit of agriculture and industry (especially shipbuilding), but nowhere enough to compete with heavyweights such as Germany.
For those out of the loop: the Balkans have historically suffered from brain drain - first during the Yugoslavia era where many fled/emigrated from realcommunism, then during the wars for obvious reasons, and now simply because Germany and other EU nations pay way better and those who don't find work in tourism find it elsewhere in Europe instead. Good luck finding a nurse on the Balkans... Germany has to recruit from the Philippines meanwhile.
The fact that Croatian (and other Balkan countries') politics are extremely corrupt doesn't help much either, it's really sad.
Regarding Southern Italy: similar situation re/ brain drain, plus the added complexity of having to deal with the Mafiya.
Regarding Turkey: Turkey already is an economic powerhorse and a regional hard-power leader - the early Erdogan years showed what Turkey is capable of. Unfortunately Erdogan turned into Erdolf and investors are pretty much shying away from Turkey as a result of the instability, not to mention that Turkey is directly adjacent to the Syria cluster-fuck.
The geography of southern Italy, Croatia, and Greece place all three at a disadvantage compared to continental nations connected via road, rail, and canal. The Mediterranean is a comparative advantage that can be leveraged. The natural beauty that attracts tourism can help repatriate the talented diasporas.
The question is whether brain drain, crime, and corruption are due to incurable pathologies or symptoms of transient disadvantages.
I've long held the conviction that tourism is a toxic sector. If it grows too large a share of GDP, so much talent, money, and effort is sucked into tourism and away from society which otherwise would have found better use for it.
Who builds the next startup, starts a franchise chain or scouts investors to build a new machine, if you can always double your salary by serving rich foreigners?
> I've long held the conviction that tourism is a toxic sector.
It's sort of a Dutch disease [1]. I've seen it happening from afar to Barcelona, which was on route to become what Berlin now is in terms of IT/programming back in ~2005-2006 but the ever increasing rent prices caused by tourism put an end to that (plus the 2008-2010 crisis, of course, which hit Spain especially hard). I had expected the same thing to happen to Amsterdam, but it looks like it managed to hold up better.
Barcelona: Particulary sad. As a tourist one could sense the unworthiness of this proud city going thorugh this transformation. From something that stood on its own feet (rich, industrious history) and aimed at creating its own future (there are still some tech-giants left - although it feels like remnants of a once brighter outlook) into something dependent on wealthy foreigners, whether it is domestically unbearable rents or a battered public life because of agressive hawkers at day and aggressive thieves at night (which eye the tourists, but pollute the place for the citizens as well). I remember somewhere in the 2000s Barcelonians put out a map for visitors (domestic and foreign) of what kinds of robberies/con games to expect in what area. And then there was of course this guy: https://www.theguardian.com/cities/2015/aug/11/commuting-fro...
> I had expected the same thing to happen to Amsterdam, but it looks like it managed to hold up better.
Well... many of the German tourists only come to Amsterdam for smoking pot on a day or weekend trip, and the French additionally for a night in the brothels since sex work is banned in France, so all you need is a lot of cheap hotels with beds, no stuff like beach resorts or other... more high-class venues to deal with these people.
Additionally, over the last years many of the "coffee shops" (weed shops) have closed down - in the early 2000s there were 280+ in Amsterdam, now there are 166. The government wants to introduce a "weed pass" that's only for Dutch citizens to further crack down on weed tourism: https://www.tagesschau.de/ausland/corona-coffeeshops-101.htm...
> The Mediterranean is a comparative advantage that can be leveraged.
How? There isn't much trade between Africa/Arabia and Europe other than oil, some agricultural products and used cars/outright waste.
> The natural beauty that attracts tourism can help repatriate the talented diasporas.
That's already the case in Croatia, many pensioners who worked in richer European countries retire back in Croatia because they can "live like kings" from pensions that would barely fetch a 1br micro apartment otherwise. For 500€ you can get a 75 m² flat in the center of Rijeka - in Munich that would be around 1500-2000€.
And those in working age... it's rare for them to return to their homelands for that reason.
> The question is whether brain drain, crime, and corruption are due to incurable pathologies or symptoms of transient disadvantages.
Neither, in my opinion. "Incurable pathologies" is bordering on racism, but it aren't "transient" issues on the other side. What's needed is massive amounts of wealth redistribution across Europe, combined with throwing the whole lot of political elites into jail (and that's also sadly valid for Germany, just look at Andreas Scheuer or the MPs who allegedly got huge kickbacks for anti-corona masks).
Basically Europe would need something like what the US did post-1945: a complete clean-up. Absent that, I'd also accept a revolution of the masses, but that isn't on the pipelines anywhere except in France...
How? Subsea power cables, optical fiber, and pipelines (?). Midsize autonomous ships providing a cost effective alternative to truck and rail transport. Promotion of English as the lingua franca. Policies that attract new talent and promote the free movement of goods and people between new coastal charter cities. Partnering with people in the same boat (or sea).
Adam Smith not only promoted specialization but also extending the "reach" of trade. Politicians have the power to ruin things but they only succeed when riding the coat-tails of talented makers. I'd focus on promoting the makers rather than punishing past ruiners. Nihilism is never the answer.
Which is also correct for Turkey. I'm just a data-point, but Facebook showed me that nearly everyone who can get a job and a visa will leave and, IMHO, not just because of Erdoğan. He was the reason I left but I decided to stay in Germany permanently for other reasons, and those other reasons are probably more clear to others now even before leaving the country.
Related: Turkey is not an economic powerhouse at all. You can't have such a fragile economy and still be called that. You think Erdoğan keeps poking at sensitive matters because he has power to do so? Those are just distractions.
Another parallel between our countries is that we both have "centrists" in power that are actually rather on the far-right... well, that happens when all young and progressive minds leave for greener fields. :(
You have two major political parties that have vied for power since the war, and only one of those of is of the right (HDZ).
And it has little to do with "progressive" minds, and everything to do with opportunity. I personally have known Croatians of all politics who have left the country due to lack of opportunity, not because they were progressive.
Also, leaving the country because of lack of opportunity becomes a self-fulfilling prophecy.
> In the end, code doesn’t make software – people and communities do... It’s hard to beat an incumbent, and even harder to do so without having a large target user community you can capture with a compelling use case.
We need a better name for the category of Machine Learning Workbench tools that include Julia, R, and NumPy+ (the ML ecosystem that uses Python as an Internal DSL). The compelling use case for Julia is that it is a modern language built on top of the LLVM platform (like Rust and Swift). There is no shame in not winning when you were late out of the gate; see the story of ARM. The value proposition hasn't disappeared it just didn't outpace the evolving bricolage of NumPy+ and its new complementers (Jupyter, PyTorch, TensorFlow).
This article by the author/podcaster Seth Godin [1] highlights the issues with Non-Fungible Tokens (NFT) [2], built on the Ethereum cryptocurrency. It extends the ideas introduced in an Everest Pipkin post [3]:
> Cryptocurrencies and NFTs are an absolute disaster for so many more reasons than the ecological.
The ecological issue is due to the energy consumption required to perform (some^) cryptocurrency computations.
Except there are non-ETH NFTs which don't consume much electricity and the author is specifically talking about some of them - like NBA Topshots - while pretending they all have large consumption.
0data appears to be a curated collection of five best-in-class libraries/frameworks that enable building SPA apps with user controlled data, rather than Big Tech controlled platforms and services.
We are developing both server and apps for [3] https://solidproject.org/ The effort is backed by Tim Berners Lee, with great auth libraries including dPop and PKCE token exchange
If it's a web app, you can check whether it's sending data anywhere but your Pod - but of course, by then it's too late.
In practice, Solid is only the technology that enables control of your data; it doesn't provide the incentives. It's still up to apps to actually provide you with that control, and reasons for it doing so can include customer demand or regulatory pressure.
But Solid can't force apps to give you control - after all, they might just choose not to use Solid in the first place.
So these services/libraries are meant to be included in apps to provide this capability?
The solution, help, I'm looking for is to be able to easily integrate this into development - to give users control of running the apps with their data but also for the ability to give access/send the data to a centralized platform - or request/command for it be removed; obviously that decision will be if they trust the platform, the governance and leadership of it.
I'm not sure exactly what you mean by "these services/libraries", but yes, in Solid, apps need to be explicitly built to support Solid, and data can only be stored in Solid Pods.
Although it's not Solid, an older project is https://remotestorage.io, which is a library that needs to be explicitly used by apps supporting RemoteStorage, but does allow users to store the data on Google Drive or Dropbox too, IIRC.
Why is Solid architected such that your data resides on a computer that you don’t own, when people own computers and phones that are plenty capable of storing data and communicating via the Internet?
There are a number of reasons you might prefer an architecture like this. The main two are resilience (don't place all your eggs in one basket -- if your computer is damaged or lost you don't lose your data) and availability (if your computer is shut down you can still access your data from your phone, tablet, or another device).
It would make great sense to have a non-hosted backend option, though, for power users who prefer to self-host their data.
Resilience and availability are two great reasons, but even then, why not get those by utilizing end-to-end encrypted sync, such that the server does not have access to the data, but the devices do?
Decoupling data storage from the application layer has some advantages, yes, but keeping the data in the clear server-side brings along many of the problems of the original:
- Data is still centralized in large repositories maintained by a company, and these repositories are still valuable and still open to attack, from both inside and outside of the company.
- There is a pinky promise between the data storage company and the consumer to treat their data properly and not to look at it or sell aggregated versions of it, but it is, at best, a pinky promise.
I agree that the open nature of the protocol making "self-hosted" an option is absolutely fantastic. But until that is accessible and easy for every single person to use, then only smart tech people will truly "own their data". That's really all I'm proposing: self-hosting being considered the default.
I'm not sure how e2e encrypted sync works in this context. Someone needs to own a key. If the app is open, then it can't be an app key (or else it doesn't protect against bad actors on the server). Is there meant to be a tertiary party that holds the key then? Probably it would be difficult for most normal people to use a product like this.
Apple syncs iMessages and Health data between a user's devices in an end-to-end encrypted manner, and plenty of normal people use those apps. They don't even need to be aware that those are end-to-end encrypted.
Imagine the device has a (non-encrypted) database, and the app runs locally and interacts with that database, like normal. Think localStorage if you are web-oriented.
The sync would be a separate background process (i.e. managed by the "Solid server" part) that handles encryption and decryption. As for how to manage a "circle" of devices that share the key without revealing it to the server: you can add a device via a key-exchange with the untrusted device asking a trusted device for the key. You can perform a key roll to remove a device. This can all be done automatically, though, where all the user sees is a control to add or remove a device. The hard part is key escrow (you throw all of your devices in a lake), by password protecting a copy of the key. Apple uses HSMs and Signal uses SGX to prevent brute-forcing this backup key.
I guess this is a design decision. Some services might be ok with the "if you lose your password you may lose everything without recourse" approach. I think it would be a tough sell for a lot of customers. Is iMessage really handled this way? I can believe that Signal is, considering how often they remind me to remember my PIN.
I said this in another comment. A small open source project I love is https://Nomie.app. Every iteration has been made so the creator can’t see any data. It has been handled well enough that two friends were able to set it up before. Though they used hosted couchdb services to store their data.
Yes, Solid comprises open specifications and anyone can implement it. The open source implementation that sees most development atm is https://github.com/solid/community-server, which is nearing a 1.0 release.
Perhaps so. I don't know the details. I was operating under the assumption that the parent comment was correct in its accusation that there was no self-hosting option.
Most users will likely prefer a secure way to sync data. There should be a self hosted option.
I am a huge fan and even started looking at Svelte to contribute to https://nomie.app. It is going back to syncing with your own CouchDB instance or just locally. Right now it’s local or to some blockchain company. I don’t remember the exact details any more but it seemed like no one had access to the data.
With Devonthink for Apple ecosystem, I synced between LAN/bonjour only for a while before switching to iCloud sync this year.
You can run the pod service on your own computer as well or on your own VPC in any cloud. Our TrinPod Server can be run on any ubuntu machine anywhere, we plan to get it to install on Raspberry Pi too. The Solid protocol is a peer to peer protocol. You just need to have 80/443 open to collaborate with the rest of the world.
> NVDLA is available for product development as part of NVIDIA's Jetson Xavier NX, a small circuit board in a form factor about the size of a credit card which includes a 6-core ARMv8.2 64-bit CPU, an integrated 384-core Volta GPU with 48 Tensor Cores, and dual NVDLA "engines", as described in their own press release. NVIDIA claims the product will deliver 14 TOPS (tera operations per second) of compute under 10 W, but most of this likely comes from the GPU cores. Applications broadly include edge computing inference engines, including object recognition for autonomous driving.
> NVIDIA's involvement with open hardware includes the use of RISC-V processors as part of their GPU product line-up.
See Ian McGilchrist’s 2009 book The Master and His Emissary:
> McGilchrist digests study after study, replacing the popular and superficial notion of the hemispheres as respectively logical and creative in nature with the idea that they pay attention in fundamentally different ways, the left being detail-oriented, the right being whole-oriented.
Without listing the actual misconceptions some people have, it's hard to say 'the left brain/right brain divide' is a misconception since I was already aware of this understanding that they just observe in different ways, not that "left brain is creative and right is logical". I grew up with the internet, though, so my understanding didn't solely originate from grade school science books that might be simplifying this phenomenon.
Each time I lose feeling in right side of body, and lose the ability to write. Not 100%, but close enough. Typing is fine however, if slow. Critical thinking goes to crap. Talking is fine. I can still draw, though my motivation goes to crap.
Is weird what stays and what goes.
Last time I looked writing is a left brain activity. Which controls right side.
Usually the right hemisphere is linked with comprehending speech/writing, rather than producing it, so it's interesting that you lose the ability to write. Is your speech affected at all? It could be because writing involves a bunch of other more domain-general brain functions too I guess.
SQLite is ubiquitous as an embedded database in client-side apps, especially iOS and Android. This ubiquity and simplicity make it a viable alternative to PostgreSQL during development. This is why all popular ORM/QueryBuilder frameworks support SQLite and fits with your appreciation of it.
SQLite as an embedded server-side database requires extra work and configuration to make it a viable alternative to PostgreSQL. It lacks good write concurrency and recoverability by default. It is, however, continually improving but gaps remain.
Since it does not have a wire protocol, SQLite is rarely connected to a data warehouse via ETL so it does not fit well as an alternative to TiDB.
> SQLite as an embedded server-side database requires extra work...
No, it depends on the server application. An all-read or read-mostly server requires nothing special. Same for any server that expects a low number of users or database per user(s).
Mozilla uses it on servers for documentation sites. You can also run your own Firefox sync server using SQLite.
Traditionally, many OLTP operational databases are connected via ETL to an OLAP data warehouse; they are not mutually exclusive. PingCap is the company behind TiDB, an HTAP NewSQL engine that competes with CockroachDB and YugabyteDB; the OP is content marketing for TiDB.
TiDB distinguishes itself with HTAP; transparently incorporating OLTP/ETL/OLAP in a single cluster. You have to specify the ETL layer and data warehouse in addition to PostgreSQL to make an apples to apples comparison; that is the core of HTAP positioning.
SAP HANA is the poster child for HTAP, a data warehouse with good enough OLTP performance to replace Oracle RDBMS; a single system is used for both SAP app tiers, Business Suite and Business Warehouse. The same value proposition applies to cloud apps. Independent OLTP/ETL/OLAP is still robust and is more modular while HTAP is more tightly integrated and simpler to operate.
We tried to deploy HANA with BW in a larger-ish life sciences company (200k employees) and so far it was a huge waste of money. It's not actually performing well and the support team had to stop replicating some of our most important data becase "there's too much".
I'm not convinced HTAP can actually work - the they OLTP and OLAP works internally seems too different.
That’s interesting and makes sense. SAP introduced HANA NLS (nearline storage) based on Sybase IQ to address your use case (I think). HANA HTAP is in-memory while IQ works with shared storage clusters so it’s ideal for offloading massive amounts of historical BW data that doesn’t fit in memory.
In-Memory HANA is freakishly good for running OLAP BW queries against fresh data. Column stores like HANA and IQ are both good at this, but to be honest, I don’t know how BW systems were typically configured before HANA/IQ.
I have to admit I'm not super involved with the system in question, and it's possible the team is underfunded or just not that great.
But we definitely ended up in a situation where it was hard to get to some important data, since the team couldn't SLT it to BW because "it's too many transactions" and trying to get it from Sidecar was blowing up, because it's too much data. And that was already with S/4.
But again, I don't know why it was the case. It would have been great and saved us a lot of work if it worked out and we could've done stuff in HANA directly, instead of copying data to different OLAP system daily. Especially now, when everyone is trying to get on the realtime-train.
I found this defense of Myers Briggs Type Index (MBTI) against the Big Five [1] OCEAN personality traits helpful. Neuroticism is a useful measure but has negative connotations that requires self-disclosure and, therefore, less likely to be shared with co-workers and acquaintances.
The advantage of MBTI is the value neutral labels that encourage sharing and discussion amongst laypeople. This is exactly why it is popular in corporate settings.
> [Arrow] adoption will take time, and most people are probably more comfortable seeing NumPy arrays. Therefore a Vaex version 4 a DataFrame can hold both NumPy arrays and Apache Arrow arrays to make the transition period easier.
There seems to be agreement that Apache Arrow is the future of dataframes across ML ecosystems. I didn't realize this transition impacted NumPy arrays in addition to Pandas dataframes in Python.
[1] https://vaex.io/blog/a-hybrid-apache-arrow-numpy-dataframe-w...