You’re correct that code migration is still a to do, we’ll be exploring how to do that in the future.

As you can imagine there are a lot of challenges with it, but ideas are welcome!

It seems like generating some data structure definitions (similar to a protobuf) and checking them in along with the code would be useful? Then the compiler can tell you when they still match, or you made an incompatible change to the code.

This seems like a situation where defining the data structure in two different ways might be good?

This seems like a good use case for Cap'n Proto.

Yes, we planned to update the coroc compiler to generate this value instead of having it hardcoded so it will be more future proof.

This is a good callout, although we probably won't be able to significantly improve the performance of Go compiled to WASM until WebAssembly evolves and introduces support for threads or stack switching so we can define goroutines based on those; right now the main reason Go compiled to WASM isn't in the ballpark of native perf is due to the stack switching emulation we have to do to get the cooperative scheduling of goroutines to work. We'll need WASM runtimes to offer more advanced primitives that we can rely on to implement those features of the Go language and produce much higher-quality code.

Would the emulation penalty still occur if Go routines aren't used at all? I have many small domain-specific libraries that I am planning to port to wasm. These libraries only allocate dynamic memory and differ anonymous functions.

This is a very interesting project, thanks for pointing it out!

Thanks for the support!

Note that timecraft runs programs compiled to WebAssembly, so the "guest" application that runs on it does not have to be written in Go!

That being said if you're interested in contributing to the project, that will be in Go.

Don't hesitate to reach out on Github if you try it out and have any questions or feedback!

Is source level debugging a possibility with this or does it only cover breakpoints on .wat text?

Source-level debugging is what we have in mind; we have done a lot of work on Wazero and as part of https://github.com/stealthrocket/wzprof to enable it.

It is still not a solved problem, and it requires support for each programming language that does things differently, but we're working towards this.

Thanks for bringing up those questions!

We did not make mention of the performance overhead because we don't have that data to share yet. Timecraft is a young project, and our initial focus is on exploring the use cases that the solution applies to.

That being said, we know that performance is extremely important, and our engineering team has spent a lot of time in their career working on scaling and optimizing large-scale and high-performance systems. Performance will be part of the challenges on our journey, and we're excited for when the time will come to focus on it!

As soon as we have metrics from production workloads that we can release, we'll publish them to share our learnings with our user base. Until then, stay tuned!

You're not wrong, software development is costly, and we have a team of bright and experienced engineers tackling this problem!

There is a lot more to timecraft than improving the debugging experience, and we tried to put into perspective some of the opportunities we see in the README.

If you happen to try it out, let us know what you think about the tech!

In case it wasn't super clear I meant like "RAM", not like, human memory?

My bad, I misread your initial comment as "it would cost a lot of money"!

Memory might be the limiting resource to support time travel debugging in some use cases, but for the ones we are targetting, the problem space is a bit more diverse. When multiple machines and multiple languages are involved, and web services are running online on a distributed compute fleet, the variety of challenges to account for goes beyond the amount of hardware that we can throw at the problem.

Correct me if I’m wrong, but MQTT is just a network protocol, it doesn’t solve storage, retries, failure resilience, etc... it’s well suited for pub/sub operations over TCP, not so much for ensuring “exactly once” delivery of messages.

Yes, you are correct. MQTT is a network protocol.

You build your infrastructure on top of it to support stuff like storage, retries, resiliency, etc...

However, it does gives you some guarantee, namely that the message will oblige its QoS (at most once, at least once, exactly once).

On top of these guarantees, you build whatever you need.

Honestly, I am quite glad that you are writing these posts, it is a great service for the community and if you end up open sourcing it, the project will bring a lot of value to everybody.

Thanks for your posts :)

Most MQTT implementations support "exactly once delivery": https://en.wikipedia.org/wiki/Comparison_of_MQTT_Implementat...

There are no issues with memory utilization of the MySQL databases other than once the working data set don’t fit in memory anymore it has to be fetched from disk on cache misses, which slows down fetches. Disk utilization of the Job databases increases over time because we don’t do any deletes to avoid extra write operations (a delete is basically a write). Once a database disk utilization reaches a defined threshold it is swapped for a spare, then drained, destroyed, and reinitialised to be reused later on.

I hope this gives some clarity, let me know if you have further questions.

We do want to open source the code. In order to get to production we took some shortcuts which tightly integrated some components of this system with Segment’s infrastructure and would make it hard for anyone to deploy. We discussed open sourcing the code along with this blog post but we felt like it would have more value once we’ve put a bit more work into it to make it easier to work with, and add a couple of features we have in the pipeline.