When a company puts up a billboard or an ad on the bus, they don't care if the ad is seen by dashcams and dogs. All that matters is impact on the bottom line.

We at engineering decided to filter out bots. Figures fell dramatically by more than 50%.

In less that a day business mandated us to remove the filter.

Bots are real people after all

Why not use Microsoft Clarity?

> The bots are getting creepily good at mimicking engagement.

You would think Google Analytics would help a lot with this, but they seem to not care.

Too often I have been working on ecommerce gigs where the traffic comes from internal 'SEO' tools. Normally the person in charge is marketing, not technical, so it has always been difficult to get past identifying the problem to fixing it.

Often ecommerce companies are very siloed, so one person in one part of the 'team' is scraping the site with their special tools, only for another person to be doing another scrape with their special tools. You can have the guy doing the newsletter doing his thing, the guy doing organic search doing their thing, the guy doing paid ads doing their thing and someone in sales doing their own thing.

The sad thing is that they are typically just a few SQL joins away from exactly the data they want in a format they can digest. However, due to silo-ing, it can be hard to have that conversation.

On top of that, you do get new bots that need to be dealt with. The Huawei bot will scrape everything yet the store might not be delivering to China. So there are legit bots not doing ad fraud that need to be dealt with.

What I also find interesting is that nobody is interested in the server logs. They come for free, and, although not having CDN cache hits, they still record checkout transactions and any pages that don't have a freshly cached page.

Ad fraud also goes on in companies. I worked for a very successful company once and we only measured sales and what was out of stock. We didn't need open rates for emails or click through rates, our main problem was selling too much, which was a nice problem to have. Note that if you sell too much then you aren't going to get it all out the door in a timely fashion, or you run out of big lorries to put the orders in.

Since then I have not worked on a site that is as successful. Instead we have people getting praise for 'false metrics'. Anything an SEO person creates or a marketeer measures will always have some nonsense aspect to it. Or the accounting is mixed with brick and mortar sales even though free shipping and discounts have been given on each sale, with adwords used to get people through the 'door'.

Since sales manager has to report to someone on the board, if the sales numbers aren't good, the nonsense stats can be used to obfuscate the facts. The board only ever care about profit, so I don't like the way this goes down with false metrics of nonsense.

Another fundamental problem in ecommerce is a lack of basic salesmanship. If you work the shop floor doing specialist sales where you have to listen to the customer's needs, then you gain experience in the art of sales. You haven't got to be good at it, in fact it can be better to know your limitations, for me that is big ticket items where I don't have the product knowledge, however, I could always hand those sales over to a much more capable colleague.

You don't win every sale, but, in retail, you can have some really good streaks where no customer leaves empty handed. Your conversion rate is going to be more like 90% in face to face sales if you have the right product at the right price, with customers that don't buy coming back the following day or week to splash the cash.

In High Street retail there is no way you give customers 15%+ off just for stepping through the door. Yet this is table stakes in ecommerce, particularly for small to medium size shops. This instantly devalues the product.

Often there will be chatbots for whatever reason, and I am sure the likes of Dell can get that right, but your typical small ecommerce site will fluff this up too, so any customer daring to use the chatbot will not get instant help from a sales person.

So what to do?

It depends on your product, however, the goal is to get customers for life, not to churn through them. To achieve this it comes down to product, price, availability, shipping times, customer service and incentives for the customer to advertise for you, with reviews, word of mouth and all that hard stuff that needs real human skill. Sometimes it will only be a one-off sale, for example, if someone is buying a mattress. But, even then, the customer service basics matter.

What is also silly is how, with everything you buy, you will get adverts and incentives to buy what you have just purchased. I don't know why this is not considered career limiting, but nobody seems to have fixed this.

Next time, I will find out what data everyone needs and have my own script to collect just that data, then have a live backup that can be used for all internal purposes such as report generation. There will be no mystery script containers downloading 150 scripts with cookies, outside of developer supervision. My hunch is that the only numbers that really matter are sales.

As for why the sales aren't happening, so long as functionality is as it should be on all devices, then you have to dig deeper, to the knowledge level that only someone that has worked the showroom floor understands. Really, a website should be the sales person's knowledge condensed into HTML. Yet nobody asks the guy serving customers what they upsell or what product they recommend if a customer baulks at a given product. Instead we have mystery-meat AI scripts that manage these things.

Thanks for the heads up on ad fraud, normally I am a long way from that due to internal scraping efforts being far more damaging to website performance, and, as a developer, page load times matter to me far more than how much was paid for the traffic.

The word "revenue" above doesn't sound right. Did you mean "ad impressions", "clicks", "visits", or something else?

If the bot traffic is giving you 40% more revenue, well that's excellent news, right? If it's revenue, then keep doing it.

How does the advertising money get to them to make it worth their while to run bots at such a scale?

I mean sure, I suspect that people in the advertisers are doing it, but surely thats massively risky. there must be a grey market for this kind of transaction?

In Google Ads you can just turn off the option to run your ads on non-Google sites; I think it's called their Display Network. Just run your campaign only on Google's search pages.

I'm surprised the article doesn't mention this rather common solution.

I've nicked that sentence to use as the title above, since it's more neutral and representative of what the article is about.

With that said, what do you think the error is here? Could your script have had some false positives? Enough to move the dial?

I was never really a punk-rebel kid, but a certain part of me rooted in the optimism of the early-internet kinda wants to see ad-models crash and burn.

Even advertising "working normally" always had a psychic odor of exploitation and deceit. Ex: "You absolutely need this product or else your peers will hate you."

> During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic.

> Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.

I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like.

All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots.

The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots.

I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.

This is not likely to distort the basic numbers in your story, but it makes the premise questionable

If your script can correctly segregate bot traffic from human traffic, and website operators can conclude that bot traffic does not make purchases, then what -- you still don't want to blackhole that traffic right ?

  One rep I had known for years finally admitted the truth off the record. "Dude, we know," he said. "Everyone knows. But if we filtered it all out properly, our revenue would drop 40% overnight, and investors would have a meltdown."

The irony of complaining about bot traffic with bot copy is too funny.

> It was not a sophisticated piece of software, just a tool designed to observe how "users" actually interacted with the website. I was not just counting clicks; I was watching behavior

I'm working right now on an inventory management system for a clinic which would really benefit from pulling the prices and availability from a very specialised online shop. I wish I could just get a large, fully cached status of all items in a json/CSV/whatever format. But they're extremely not interested, so I'm scraping the html from 50 separate categories instead. They'll get a few daily bot hits and neither of us will be happy about it.

If people are scraping data that you're not selling, they're not going to stop - just make it trivially accessible instead in a way that doesn't waste resources and destroy metrics.

The second is best "blocked" by doing nothing at all and instead silently dropping them from analytics.

Lawsuit time?

They had plenty of other problems, including an unworkable business plan, but maybe they would have had more time to pivot before selling out for pennies if they'd not been chasing their tail so much.

You know, the kind that existed before Google created their thing.

Despite the “LinkedIn influencer” writing style of the article, the results don’t seem that shocking or unexpected.

Unlikely?

> just name them Does Adsense even gives you information where exactly your ad is getting published?

That feels a lot more like 'fraud' than 'the world evolving'

>Can you just tell e.g. Google Ads that you don't want to pay for certain clicks?

No

Sure, a lot of your visitors were handed a flyer by google before they showed up in your store, but were any of them new customers?

Facebook Ads and SA360 both allow 3rd party impression and click tracking. Not to mention the myriad of 3rd party analytics tools you can use to track the website.

> if you look at IP from their "clicks", you see often a FB/Google datacenter IP range

I've never heard this before and seems hilariously simplistic. These megacorps don't have VPNs?

> they did just clever algorithm and heavy profiling to stole your organic traffic

Not sure if you had a stroke while writing this, but this makes no sense.

As someone who has also been in the industry for 10+ years, none of what you said passes the smell test. Maybe after 15 years you still never understood the industry or technology?

I think the history of corporate bad behavior has demonstrated that any time you say "Surely, company X wouldn't dare to do [some profit-making thing]" you're probably wrong. This applies to all companies, not just Facebook. Revenue = R, Cost of the activity = A, Probability of being caught = P, Likely fine = F. If R > A + P x F, you do the thing.

Advertisement has always been largely a waste of money, the problem is always trying to figure out what part is a waste and what part is effective. Internet advertising promised to be more specifically targeted and attributable than previous advertising models, but I don't think that really turned out well.

As an advertiser, you really have to work hard to know how things are working. Ask customers to tell you how they found you, but know that customers don't always know and don't always share what they know. Adjust advertising campaigns (including turning them off) and see how things change, but changes take a while to filter though. And you've got to do all the other things too --- a lot of new or smaller companies make the mistake of spending a lot on advertisements to drive traffic to them when they can't service the traffic: coupon campaigns that get too many people in, or sending you to a page that's unclear or difficult to use so people bail out, etc.

The article claims 50k visitors, 47 sales, $4000 in ad spend; it's not really clear if that's overall visitor numbers or just from the traffic attributed to advertising. Measuring success by visitor numbers or sales counts was never the right way to measure success; the numeric measure of success for a business should be an accounting measure of net income [1], which is only somewhat related to visitor and sales count. If this business generates $500 of net income for every sale (excluding advertising costs), then $4000 in ad spend to generate 47 sales seems reasonable; if they generate $1 of net income for every sale, then $4000 in ad spend is highly problematic.

[1] GAAP or adjusted as appropriate to the business

That or dark patters that get people to click on ads in the middle of content, where the user instantly backs out once they realize.

Tried to keep going, but "a strange, unsettling feeling crept in," yes indeed.

Also please note OP is marketting copy for an analytics product that presumably says it will filter out bot traffic better?