I find the business model fascinating and it is amazing to watch them scale. My family business used to spend over $20k per year on YellowPages ads which drove about 8-12 calls per day at a very high close rate. As YellowPages has died off the money has been invested in an online presence but the skill required to design the site, keep it updated, and to manage all the online advertising spend is well beyond a normal person.
The fragmentation impact of having your customers looking for your services in dozens of different ways vs opening the phone book is costly and complex to overcome.
Once you pay a designer, hosting provider, and then pour the rest into Adwords it is very hard to get the same 8-12 calls per day with $20k of annual spend.
Thumbtack are helping solve that problem and syphoning all the money these businesses used to spend on YellowPages or are wasting on poor Adwords bidding strategies. Paying per lead is not that much different from paying $20k to YellowPages who give you no guarantee you will ever get a phone call.
8 or so leads a day is roughly ~2700 leads per year. A $20k budget means you can spend about $7.50/lead. Depending on your niche that is likely going to be difficult to do with adwords alone.
Again depending on your niche, Thumbtack might actually be good here. In my experience though Thumbtack is very hit or miss from the businesses side of things. If you're spending $5-$10 just to send an email to single potential customer it might not be cost effective.
meanpath.com can do around 200 million pages per day using 13 fairly average dedicated servers. We only crawl the front page (mile wide, inch deep) so the limiting factor is actually DNS. Looking at the network traffic the bandwidth is split evenly between DNS and HTTP. Google public DNS will quickly rate limit you so you need to use your own resolvers (we use Unbound).
Unlike Blekko we are just capturing the source and dumping it into a DB without doing any analysis. As soon as you start trying to parse anything in the crawl data your hardware requirements go through the roof. parallel with wget or curl is enough to crawl millions of pages per day. I often use http://puf.sourceforge.net/ when I need to do a quick crawl
"puf -nR -Tc 5 -Tl 5 -Td 20 -t 1 -lc 200 -dc 5 -i listofthingstodownload" will easily do 10-20 million pages per day if you are spreading your requests across a lot of hosts.
We used djbdns on every crawl machine, and did not find DNS to be limiting at all. You should also make sure there isn't any connection tracking, or firewalls/middleboxes which are doing connection-based anything, or NAT, or really anything other than raw Internet between you and the Internet.
When I click on this link, I am presented with a "sign-up" page.
"""
YOU'VE REACHED A SUBSCRIBER-ONLY ARTICLE.
Would you like to become a subscriber for 50% less for the first 12 weeks?* Sign up now and access the full breadth of The Australian's content in minutes.
"""
I have used a few of the Kimsufi boxes and had no issues with them. At meanpath.com we use 40 SoYouStart servers and 5 big OVH servers and have never had any issues. Excellent support was provided when we did run into problems with quick hardware swap outs or whole server replacements.
If you want to test out the speed of the network feel free to download some of our publicly released crawl files on http://archive.meanpath.com which is on a 2gbps burstable pipe.
Based on the fact that he got the email 40 minutes after his query to Stripe I doubt any of these companies were involved.
To do a crawl across a meaningful percentage of the Internet and refresh this every 24 hours is hard enough even if you only capture the front page. Very few sites include stripe.js in the source on the front page. So you would need to crawl deep enough into a site to hit the checkout page and capture stripe.js in the head section which is exponentially more difficult.
Indexing your crawl and passing this off to a team to extract contact details and send it to a client so they can reach out all within 40 minutes I would say is nearly impossible. It is possible they already had his details but I cannot see anything on his site that would trigger a "likely to switch payment gateways" alert.
My guess is this was completely random. A good sales rep will make 60-100 outbound calls per day along with a bunch of emails. Multiply this by the number of outbound sales reps in the payments market and you have a lot of sales activity.
Running this from any web host will almost certainly get your server turned off along with a pile of abuse emails. To make it run even faster you should scrub all the Bogon routes on this list:
https://www.team-cymru.org/Services/Bogons/http.html
I know it is not quantifiable but the effectively free marketing and goodwill that Twitter get from open sourcing Bootstrap must be worth many millions.
That "get" will quickly become a "got", as they've finally moved away from github.com/twitter to their own independent github.com/twbs repository, as was the plan for a long time:
http://blog.getbootstrap.com/2012/09/29/onward/
The good thing about Bootstrap is that it is stamped on pretty much every page of a site. Our sample index was mainly front pages so it was easy to spot. Total size of the web is a very hard number to pin down. Verisign quote 252 million base domains worldwide across all TLD but we have some base domains in our index with over 1m sub domains.
The fragmentation impact of having your customers looking for your services in dozens of different ways vs opening the phone book is costly and complex to overcome.
Once you pay a designer, hosting provider, and then pour the rest into Adwords it is very hard to get the same 8-12 calls per day with $20k of annual spend.
Thumbtack are helping solve that problem and syphoning all the money these businesses used to spend on YellowPages or are wasting on poor Adwords bidding strategies. Paying per lead is not that much different from paying $20k to YellowPages who give you no guarantee you will ever get a phone call.