The real-time syncing immutable, append-only log that hypercore provides can also be accessed randomly, allowing for lots of cool parallel/distributed log processing architectures (Kappa architecture), similar to Kafka (which is also based on immutable, append-only logs). We have just focused on syncing a filesystem at first because we had a strong use case there, but you could totally build a CRDT on top of an append only log.
Hey Max, long time no chat since Hash the Planet! Was it you or Mathias that demoed running an Operating System while streaming it live from remote peers? Those were some insanely sick demos.
Is random access something you guys have added since then? Or, could you clarify how it reduces the overhead on apps that would have shared mutable state (even if it is composed of immutable streams)? Don't you still have to back-scan through the log to recompose the current view? Which then wouldn't scale for read performance. Or is there an alternative approach?
This was one of the main things I was discussing with Dominic Tarr about Scuttlebutt back then. We had done a lot of event-sourcing (immutable logs) at my previous startup, but had problems with compaction or generating views/state of those streams. Which is what lead me to the CRDT approach as the base, not the thing on top. I know Tim Caswell/Creationix is using DAT for one of his products he is building with @coolaj86 .
But that was 4 years ago now, that me and Dominic were talking about those problems? Does random access solve that? Would love to read about it, shoot me some links! Also, Dominic and I are starting to have Web Crypto meetings with some folks at MIT (anvil/trust, ex-riak/basho) and they are in town (the Bay Area) this week. If you are around, we should all get together again to discuss identity, security, crypto, and P2P!
Great analysis. We anticipate that in order to fix these three usability issues around trust we will need to provide a centralized identity provider in the future. This would also address privacy issues especially regarding leaking what dats you are accessing. The design philosophy around Dat is to start from the end of the completely decentralized spectrum but be flexible in letting the application choose the tradeoffs as they move more towards centralized components.
It would be good to figure out key rotation for Dat URL providers since this probably has to be built into the protocol.
Any thoughts on integrating with keybase? I like keybase's model where you have device-specific keys. But this would probably make moving a Dat URL provider to a different machine trickier.
This all assumes that well-known Dat URL's become an important thing to preserve (they are published in papers, etc) even though they are very user-unfriendly, even more than IP addresses.
A naming system on top of them would make key rotation a non-issue (rotate Dat URL's instead) and you could completely replace or remove the history, sort like a git rebase. But that loses other nice properties of the system?
I suppose irrevocability is something we deal with in git repos all the time. Although you can do a rebase locally, once a commit is accepted by a popular project, they're unlikely to let you remove it from history. The review process makes it unlikely that any really embarrassing mistake would be accepted, so this seems ok in practice.
Hi, dat maintainer here. I made the website in a couple of hours (https://github.com/maxogden/dat-data.com/commits/gh-pages) but have spent about 9 months on the code so far (read through e.g. closed issues for an example of the scope there). I also have tried to convey that dat is still pre-alpha and that people shouldn't use it unless they wanna get involved w/ early development, but someone (not me) decided to post it to hacker news nonetheless.
not yet, but keep an eye on https://github.com/jbenet/transformer which will provide the data format conversion API. It would be really cool to see some sheetjs stuff hooked up to dat/transformer!
Hi, maintainer of dat here. We're working on the first release now. The first use case is tabular data (a large volume of small rows), but we'll be adding a content-addressable blob store (for big binary attachments) on top for the beta, as well as some fun p2p (webrtc data channel) powered sync stuff (will be somewhat similar to BitTorrent sync) to maximize download speeds and let universities seed big scientific datasets.
great work so far in an area I'm really interested. I'm especially interested in the synchronisation and transformation modules, how are they coming along? sync is an issue that distinguishes you from the work of the libre api project, but did you look at that project at all for the sharing aspect because I think they do that really well (but sync they do still with kill and load). also, did you look at all at git assistant for sync?
Sorry about the tone, I definitely could have said that part more nicely. To me Java smells like too many abstract interfaces, lots of boilerplate and hard to use build tools. Java was the first language I learned so I may very well be scarred from the experience.
