Likewise. I have a nasty feeling that most AI agent deployments happen with nothing more than some cursory manual testing. Going with the ‘vibes’ (to coin an over used term in the industry).
I can confirm this after hundreds of talks about the topic over the last 2 years. 90% of cases are simply not high-volume or high-stakes enough for the devs to care enough. I'm a founder of an evaluation automation startup, and our challenge is spotting teams right as their usage starts to grow and quality issues are about to escalate. Since that’s tough, we're trying to make the getting-to-first-evals so simple that teams can start building the mental models before things get out of hand.
A lot of "generative" work is like this. While you can come up with benchmarks galore, at the end of the day how a model "feels" only seems to come out from actual usage. Just read /r/localllama for opinions on which models are "benchmaxed" as they put it. It seems to be common knowledge in the local LLM community that many models perform well on benchmarks but that doesn't always reflect how good they actually are.
In my case I was until recently working on TTS and this was a huge barrier for us. We used all the common signal quality and MOS-simulation models that judged so called "naturalness" and "expressiveness" etc. But we found that none of these really helped us much in deciding when one model was better than another, or when a model was "good enough" for release. Our internal evaluations correlated poorly with them, and we even disagreed quite a bit within the team on the quality of output. This made hyperparameter tuning as well as commercial planning extremely difficult and we suffered greatly for it. (Notice my use of past tense here..)
Having good metrics is just really key and I'm now at the point where I'd go as far as to say that if good metrics don't exist it's almost not even worth working on something. (Almost.)
I'm curious what people are doing. We're still very much in the experimentation phase
> Sounds risky!
One of first attempts at building file system tools for my custom agent called `tree` and caught a few node_models. Blew up my context and cost me $5 in 60s. Fortunately I triggered the TPM rate-limit and the thing stopped
Neat! Really like this, and will be interesting to see how it tracks over time.
My only concern is the scale you have selected as we are already close to 100, are you going to have to allow it to be ‘turned up to 11’ at some point?
Very cool. I actually created a shareware application that rendered Photomosaics 25 years ago! Here is a link to "Mosaic Magic" from the Wayback Engine:
I managed to make a decent amount of spending money from that application during my University years.
One interesting lesson I learnt was about enterprise pricing. I think I charged ~$20 for Mosaic Magic, however, I started to get emails from organisations asking about pricing for commercial use. Nothing on my pricing page suggested that commercial use was prohibited, I guess they just thought $20 was rather cheap. From there-on, I charged $150 for "commercial use". Basically, anyone who thought they should be paying more, did!
Finally, Robert Silvers filed a patent and trademark for Photomosaics in ~2000, and was, for a while aggressively pursuing organisations that he felt was infringing. I assume this has all died down now.
Thanks for sharing! Both the link and the story. As a University student and someone that has here also made a photomosaic app, that is super relatable.
That's good to know about the pricing. It's free for now. Honestly didn't expect this level of interest (there's 307 people on the site right now according to analytics).
With that interest, I may add some plans - potentially for the commercial use in particular
I used Mosaic Magic back in my high school journalism class to design a yearbook cover 20+ years ago! Turned out great - it won some kind of award iirc.
Kudos to OP for not immediately thinking about how he best could monetize this.
I'm glad FOSS made Shareware die out, but now web apps requiring me to create an account first have taken its place. Kudos to OP for resisting that too.
Nice work. My first thought was whether I could get similar graphs for my own data — this run-report.com does exactly that. Clean design, and the GPT-generated summary is a nice touch. Thanks for sharing.
