I've seen the demo of this and it's pretty slick! It looks super easy to use and it uses Inngest under the hood which is awesome (I'm one of the co-founders of Inngest [https://www.inngest.com] so it's always very cool to see fun and creative projects like this launch with our product - also Sam is great!
Good question. This was a hard question for us last year and we chose SSPL for the time being as a early stage startup to offer some protection. AGPL allows anyone to deploy your system and re-sell it, but SSPL requires the person to open source their additions that they make for their platform, which benefits the project itself.
*Caveat*: This is super nuanced and hotly debated, so this is high level and no perfect answer here.
Mid term, we plan to move from SSPL to a more open license in the future as we further develop our open source project.
Exactly - we have many users that have come over after using Temporal. We designed our SDK to be more lightweight and flexible. We want it to feel more just like writing normal code, not a new coding paradigm. For example, you can define steps right within your function, not as separate "activities."
Being HTTP based (push vs. pull), it's easier to manage and works natively with serverless and servers.
Inngest is also event-driven, so you can fan-out and do things like have your workflow wait for another event. Our `step.waitForEvent()` allows you to pause a function until another event is received, creating dynamic jobs that can wait for additional actions or input. Also, using events allows us to replay failures super easily.
re: ActiveJob - Yeah, multi-step workflows are a huge difference. We manage step retries and the function state for you. That makes things like sleep and coordinating between events easy. As you mentioned, it leads to simpler function definition so it means that almost any engineer can write workflows quickly and easily read the code in a single place, reducing bugs due to disconnected jobs.
> we had rolled our own monitoring which was itself a PITA to maintain
Thanks! What type of monitoring were you looking for? We have some basic metrics now, but know we need to improve this. What metrics, alerting, observability are important for you?
Not the original commenter but I manage a similar system:
1. Wait timings for jobs.
2. Run timings for jobs.
3. Timeout occurrences and stdout/stderr logs of those runs
4. Retry metrics, and if there is a retry limit, then metrics on jobs that were abandoned.
One thing that is easy to overlook is giving users the ability to define a specific “urgency” for their jobs which would allow for different alerting thresholds on things like running time or waiting.
This is great - we do capture all logs for each run including any retries, so you can see errors and general successes. All of these other metrics we have internally, but need to expose to our users!
Observability is super key for background work even more so since it's not always tied to a specific user action, so you need to have a trail to understand issues.
> One thing that is easy to overlook is giving users the ability to define a specific “urgency” for their jobs which would allow for different alerting thresholds on things like running time or waiting.
We are adding prioritization for functions soon so this is helpful for thinking about how to think about telemetry for different priority/urgent jobs.
re: timeouts - managing timeouts usually means managing dead-letter queues and our goal is to remove the need to think about DLQs at all and build metrics and smarter retry/replay logic right into the Inngest platform.
Sorry DLQs make it easier to do those alerts where a human needs to look asap at something. Not sure they can be gotten rid of, but maybe you call them something else.
Agreed that alerting is important! We alert on job failures, plus we integrate with observability tools like Sentry.
For DLQs, you're right that they have value. We aren't killing DLQs but rather rethinking them with better ergonomics. Instead of having a dumping ground for unacked messages, we're developing a "replay" feature that lets you retry failed jobs over a period of time. Our planned replay feature will run failures in a separate queue, which can be cancelled at any time. The replay itself can be retried as well if there's still a problem
We started with TypeScript because it's not well supported with a current solution and none of them support serverless. We wanted to solve serverless first as it's made supporting long-running servers easy.
A lot of folks in the TS/JS community also don't often build distributed systems and it's easy to get wrong. So we think they're hungry for something like Inngest that they don't need to manage or spend weeks learning some complex system. Plus, TS gives us typing for all events/messages.
We already have a working Go SDK that we use internally and we have a test harness that will enable us to add other languages like Rust or Zig more easily. We even have a community member building a PoC for Elixir.
This is a great one. I've experienced this myself, especially when you change an event/message and then you need to handle that change in your job/workflow. Things can break pretty easily so you need to have versioning for both.
This is why we've built event schema versioning and versioning for functions baked into the platform. We have big plans for the schema management side of things that bring concepts of data governance to engineering teams. It should just be for data teams. As a bonus, we can also generate language types from schemas easily then.
What else about schema management is a pain? What have you used for this?
Congrats on the launch. Building a reliable background job / workflow infra is hard. Temporal has lifted the bar significantly, glad to see new development.
As for the schema management part, we at bytebase.com have also built an OSS product to tackle this specifically.
"an employee could look at that formula after a year and think, "Well, I know where I am now and where I could be in two years, no reason to stay here for that.""
The good part about this is that the formula and system is open to change and will continue to evolve as we find ways that it could be working better for us. We're actually in the midst of some changes right now that have and will really help the formula work in the coming months & years.
Yeah, I'm not at all surprised about that based on what this degree of openness says about the culture of Buffer. Keep it up.
I also have to think balancing such a formula is a very interesting challenge. On the one hand you don't want it so hard coded that there is no wiggle room, but on the other a higher fudge factor can really change the type of applicants that come along.
My personal top reasons are the culture, the people and what we value as a company. I believe it's a phenomenal place to work as an engineer and the remote working setup has been one of the most fun, interesting and liberating ways to work in my career.
Culture - I find the engineering culture great, lots and lots of trust, everyone deploys at all times throughout the day, freedom to experiment with new technologies. No one points fingers and the team is all in it together. Non-engineering wise, there is a ton of respect for each other on the team, it's a super positive place to work.
The people - you'd have to get to know them, they're awesome bunch :)
I can't help but think that there are tons of places that don't treat people like children and are respectful to each other. I wouldn't want to get underpaid for something generic like this. Just my personal opinion, everyone is different.
I just feel like you are being underpaid by a lot.
Sure! My previous employers were also great places to work, I've been pretty fortunate in my career. I can understand my response may feel generic, if we were having this convo over a coffee or beer, I could tell you in greater detail.
If I was someone just after the money, I could choose to go downtown and work in fin-tech. Everyone has their own motivations, values and desires. Buffer fits very well within mine!
How is it underpaid if you're not living in SF? Some of them are living in countries that are pretty cheap, they actually live much better and save way more than anyone living in SF.
Thanks for the post! I'm Dan, the creator of timezone the open source project and am now working on a product version at http://timezone.io. Just pushed the new homepage - Hope that helps paint a picture for whats to come there!
Until then, feel free to fork the project, and submit Pull Requests! Would love to hear the ideas you might have :)
Dan here - I created timezone the open source project and am now working on a product version at timezone.io. Didn't know people would stumble upon it yet so I just pushed the new homepage - Hope that helps paint a picture for whats to come!
