Doesn't that effectively let anyone host anything there?
It's more like sites.google.com.
Sqlite used to have a limit of 999 query parameters, which was much easier to hit. It's now a roomy 32k.
We probably should have been partitioning the data instead of inserting it twice, but I never got around to fixing that.
COPY is likely a better option if you have access to the host, or provider-specific extensions like aws_s3 if you have those. I'm sure a data engineer would be able to suggest a better ETL architecture than "shove everything into postgres", too.
is even funnier :D
>Some phones will silently strip GPS data from images when apps without location permission try to access them.
That strikes me as the right thing to do?
And wait. Uh oh. Does this mean my Syncthing-Fork app (which itself would never strike me as needing location services) might have my phone's images' location be stripped before making their way to my backup system?
EDIT: To answer my last question: My images transferred via Syncthing-Fork on a GrapheneOS device to another PC running Fedora Atomic have persisted the GPS data as verified by exiftool. Location permissions have not been granted to Syncthing-Fork.
Happy I didn't lose that data. But it would appear that permission to your photo files may expose your GPS locations regardless of the location permission.
Looking now I can't even find that setting anymore on my current phone. But the photos still does have the GPS data intact.
Yep, and it's there for very goos reasons. However if you don't know about it, it can be quite surprising and challenging to debug.
Also it's annoying when your phones permissions optimiser runs and removes the location permissions from e.g. Google Photos, and you realise a few months later that your photos no longer have their location.
What happens is that when an application without location permissions tries to get photos, the corresponding OS calls strip the geo location data when passing them. The original photos still have it, but the application doesn't, because it doesn't have access to your location.
This was done because most people didn't know that photos contain their location, and people got burned by stalkers and scammers.
Try to get an iPhone user to send you an original copy of a photo with all metadata. Even if they want to do it most of them don't know how.
Every kind of permission should fail the same way, informing the user about the failure, and asking if the user wants to give the permission, deny the access, or use dummy values. If there's more than one permission needed for an operation, you should be able to deny them all, or use any combination of allowing or using dummy values.
I don't disagree that months should be 1-indexed, but I would not make that assumption solely based on days/years being 1-indexed, since 0-indexing those would be psychotic.
For example, the first day of the first month of the first year is 1.1.1 AD (at least for Gregorian calendar), so we could just go with 0-indexed 0.0.0 AD.
I don't think adding counterintuitive behavior to your data to save a "- 1" here and there is a good idea, but I guess this is just legacy from the ancient times.
Can't wait for it to be stable and widely available, it's just too good.
> month values start at 1, which is different from legacy Date where months are represented by zero-based indices (0 to 11)
[0] https://tc39.es/proposal-temporal/docs/
[1] https://tc39.es/proposal-temporal/docs/plaindate.html#month
I've now read the entire Cursed Knowledge list & - while I found some of them to be invaluable insights & absolutely love the idea of projects maintaining a public list of this nature to educate - there are quite a few red flags in this particular list.
Before mentioning them: some excellent & valuable, genuinely cursed items: Postgres NOTIFY (albeit adapter-specific), npm scripts, bcrypt string lengths & especially the horrifically cursed Cloudflare fetch: all great knowledge. But...
> Secure contexts are cursed
> GPS sharing on mobile is cursed
These are extremely sane security feature. Do we think keeping users secure is cursed? It honestly seems crazy to me for them to have published these items in the list with a straight face.
> PostgreSQL parameters are cursed
Wherein their definition of "cursed" is that PG doesn't support running SQL queries with more than 65535 separate parameters! It seems to me that any sane engineer would expect the limit to be lower than that. The suggestion that making an SQL query with that many parameters is normal seems problematic.
> JavaScript Date objects are cursed
Javascript is zero-indexed by convention. This one's not a huge red flag but it is pretty funny for a programmer to find this problematic.
> Carriage returns in bash scripts are cursed
Non-default local git settings can break your local git repo. This isn't anything to do with bash & everyone knows git has footguns.
Also the full story here seemed to be
1. Person installs git on Windows with autocrlf enabled, automatically converting all LF to CRLF (very cursed in itself in my opinion).
2. Does their thing with git on the Windows' side (clone, checkout, whatever).
3. Then runs the checked out (and now broken due to autocrlf) code on Linux instead of Windows via WSL.
The biggest footgun here is autocrlf but I don't see how this is whole situation is the problem of any Linux tooling.
If git didn't have this setting, then after checking out a bash file with LFs in it, there are many Windows editors that would not be able to edit that file properly. That's a limitation of those editors & nobody should be using those pieces of software to edit bash files. This is a problem that is entirely out of scope for a VCS & not something Git should ever have tried to solve.
In fact, having git solve this disincentives Windows editors from solving it correctly.
Well, bash could also handle crlf nicely. There's no gain from interpreting cr as a non-space character.
(The same is valid for every language out there and all the spacey things, like zero-width space, non-breaking space, and vertical tabs.)
TL;DR - if your repo will contain bash scripts, use .gitattributes to make sure they have LF line endings.
This is just a list of things that can catch devs off guard
> JavaScript date objects are 1 indexed for years and days, but 0 indexed for months.
This mix of 0 and 1 indexing in calendar APIs goes back a long way. I first remember it coming from Java but I dimly recall Java was copying a Taligent Calendar API.
Dark-grey text on black is cursed. (Their light theme is readable.)
Also, you can do bulk inserts in postgres using arrays. Take a look at unnest. Standard bulk inserts are cursed in every database, I'm with the devs here that it's not worth fixing them in postgres just for compatibility.
Its unclear exactly what conditions cause a site to get blocked by safe browsing. My nextcloud.something.tld domain has never been flagged, but I’ve seen support threads of other people having issues and the domain name is the best guess.
https://photos.example.com/albums/xxxxxxxx-xxxx-xxxx-xxxx-xx...
Then suddenly the domain is banned even though there was never a way to discover that URL besides GMail scanning messages. In my case, the server is public so my siblings can access it, but there's nothing stopping Google from banning domains for internal sites that show up in emails they wrongly classify as phishing.
Think of how Google and Microsoft destroyed self hosted email with their spam filters. Now imagine that happening to all self hosted services via abuse of the safe browsing block lists.
https://photos.example.com/albums/xxxxxxxx-xxxx-xxxx-xxxx-xx...
That's not going to be gleaned from a CT log or guessed randomly. The URL was only transmitted once to one person via e-mail. The sending was done via MXRoute and the recipient was using GMail (legacy Workspace).
The only possible way for Google to have gotten that URL to start the process would have been by scanning the recipient's e-mail.
I've read almost everything linked in this post and on Reddit and, with what you pointed out considered, I'd say the most likely thing that got my domain flagged is having a redirect to a default styled login page.
The thing that really frustrates me if that's the case is that it has a large impact on non-customized self-hosted services and Google makes no effort to avoid the false positives. Something as simple as guidance for self-hosted apps to have a custom login screen to differentiate from each other would make a huge difference.
Of course, it's beneficial to Google if they can make self-hosting as difficult as possible, so there's no incentive to fix things like this.
Also - when you say banned, you're speaking of the "red screen of death" right? Not a broader ban from the domain using Google Workplace services, yeah?
Yes.
> I would love for someone to attempt this in as controlled of a manner as possible.
I'm pretty confident they scanned a URL in GMail to trigger the blocking of my domain. If they've done something as stupid as tying GMail phishing detection heuristics into the safe browsing block list, you might be able to generate a bunch of phishy looking emails with direct links to someone's login page to trigger the "red screen of death".
Possible scenario:
- A self-hosted project has a demo instance with a default login page (demo.immich.app, demo.jellyfin.org, demo1.nextcloud.com) that is classified as "primary" by google's algorithms
- Any self-hosted instance with the same login page (branding, title, logo, meta html) becomes a candidate for deceptive/phishing by their algorithm. And immich.cloud has a lot of preview envs falling in that category.
BUT in Immich case its _demo_ login page has its own big banner, so it is already quite different from others. Maybe there's no "original" at all. The algorithm/AI just got lost among thousands of identically looking login pages and now considers every other instance as deceptive...
I'm guessing Google's phishing analysis must be going off the rails seeing all of these login prompts saying "immich" when there's an actual immich cloud product online.
If I were tasked with automatically finding phishing pages, I too would struggle to find a solution to differentiate open-source, self-hosted software from phishing pages.
I find it curious that this is happening to Immich so often while none of my own self-hosted services have ever had this problem, though. Maybe this is why so many self-hosted tools have you configure a name/descriptor/title/whatever for your instance, so they can say "log in to <my amazing photo site>" rather than "log in to Product"? Not that Immich doesn't offer such a setting.
Normally I see the PSL in context of e.g. cookies or user-supplied forms.
Yes. For instance in circumstances exactly as described in the thread you are commenting in now and the article it refers to.
Services like google's bad site warning system may use it to indicate that it shouldn't consider a whole domain harmful if it considers a small number of its subdomains to be so, where otherwise they would. It is no guarantee, of course.
For example, if users are supposed to log in on the base account in order to access content on the subdomains, then using the public suffix list would be problematic.
I'm not sure how people not already having hit this very issue before is supposed to know about it beforehand though, one of those things that you don't really come across until you're hit by it.
Fun learning new things so often but I never once heard of the public suffix list.
That said, I do know the other best practices mentioned elsewhere
Which then links to: https://github.com/publicsuffix/list/wiki/Guidelines#submitt...
Fairly obvious and typical webpage > documentation flow I think, doesn't seem too hard to find.
Google 90s to 2010 is nothings like Google 2025. There is a reason they removed "Don't be evil" ... being evil and authoritarian makes more money.
Looking at you Manifest V2 ... pour one out for your homies.
This is the first thing i disable in Chrome, Firefox and Edge. The only safe thing they do is safely sending all my browsing history to Google or Microsoft.
This feature is there for my mother-in-law, who never saw a popup ad she didn't like. You might think I'm kidding; I am not. I periodically had to go into her Android device and dump twenty apps she had manually installed from the Play Store because they were in a ring of promoting each other.
Well, if the legal system used the same "Guilty until proven innocent" model, we would definitely "catch more bad actors than false positive good actors".
That's a tricky one, isn't it.
A better analogy, unfortunately for all the reasons it's unfortunate, is police: acting on the partial knowledge in the field to try to make the not-worst decision.
many google employee is in here, so I dont expect them to be agree with you
Google needs to be held liable for the damages they do in cases like this or they will continue to implement the laziest solutions as long as they can externalize the costs.
I wish this comment were top ranked so it would be clear immediately from the comments what the root issue was.
For example:
1. Add a subdomain to test something out
2. Complete your test and remove the subdomain from your site
3. Forget to remove the DNS entry and now your A record points to an IP address
I had this happen to me once with PDF books being served through a subdomain on my site. Of course it's my mistake for not removing the A record (I forgot) but I'll never make that mistake again.
10 years of my domain having a good history may have gotten tainted in an unrepairable way. I don't get warnings visiting my site but traffic has slowly gotten worse over time since around that time, despite me posting more and more content. The correlation isn't guaranteed, especially with AI taking away so much traffic but it's something I do think about.
This is very clearly just bad code from Google.
In the past, browsers used an algorithm which only denied setting wide-ranging cookies for top-level domains with no dots (e.g. com or org). However, this did not work for top-level domains where only third-level registrations are allowed (e.g. co.uk). In these cases, websites could set a cookie for .co.uk which would be passed onto every website registered under co.uk.
Since there was and remains no algorithmic method of finding the highest level at which a domain may be registered for a particular top-level domain (the policies differ with each registry), the only method is to create a list. This is the aim of the Public Suffix List.
(https://publicsuffix.org/learn/)
God I hate the web. The engineering equivalent of a car made of duct tape.
Kind of. But do you have a better proposition?
End of random rant.
But then you would loose plattform independency, the main selling point of this atrocity.
Having all those APIs in a sandbox that mostly just work on billion devices is pretty powerful and a potential succesor to HTML would have to beat that, to be adopted.
The best thing to happen, that I can see, is that a sane subset crystalizes, that people start to use dominantly, with the rest becoming legacy, only maintained to have it still working.
But I do dream of a fresh rewrite of the web since university (and the web was way slimmer back then), but I got a bit more pragmatic and I think I understood now the massive problem of solving trusted human communication better. It ain't easy in the real world.
This all just drives a need to come up with ever more tacked-on protection schemes because browsers have big targets painted on them.
Anyway, in your scenario the controller would be essentially a one off and you'd be better off writing a native app to interface with it for the one computer this experiment will run on.
And that's before realizing it's already a bad idea with existing devices because they were never designed for giving untrusted actors direct access.
Not unlike the programming language or the app (growing until it half-implements LISP or half-implements an email client), the browser will grow until it half-implements an operating system.
For everyone else, there's already w3m.
WebUSB I don't use or would miss it right now, but .. the main potential use case is security and it sounds somewhat reasonable
"Use in multi-factor authentication
WebUSB in combination with special purpose devices and public identification registries can be used as key piece in an infrastructure scale solution to digital identity on the internet."
You have sites now that let you debug microcontrollers on your browser, super cool.
Same thing but with firmware updates in the browser. Cross platform, replaced a mess of ugly broken vendor tools.
Your micro-controllers should use open standards for their debugging interface and not force people to use the vendor website.
Yes. Regards, CIA, Mossad, FSB etc.
You remove that, and videoconferencing (for business or person to person) has to rely on downloading an app, meaning whoever is behind the website has to release for 10-15 OSes now. Some already do, but not everyone has that budget so now there's a massive moat around it.
> But do we need e.g serial port or raw USB access straight from a random website
Being able to flash an IoT (e.g. ESP32) device from the browser is useful for a lot of people. For the "normies", there was also Stadia allowing you to flash their controller to be a generic Bluetooth/usb one on a website, using that webUSB. Without it Google would have had to release an app for multiple OSes, or more likely, would have just left the devices as paperweights. Also, you can use FIDO/U2F keys directly now, which is pretty good.
Browsers are the modern Excel, people complain that they do too much and you only need 20%. But it's a different 20% for everyone.
Same as your camera/microphone/location.
If you give USB access, it is not really a website anymore, rather a app delivered through the web. I don't see a fundamental difference in trust.
I rather am able to verify the web based version easier and I certainly won't give access to a random website, just like I don't download random exes from websites.
Performance is lower, yes and well ... like I said, it is all a big mess. Just look at the global namespace in js. I still use it because of that power feature called plattform independence. What I release, people can (mostly) just use. I (mostly) don't care which OS the user has.
But do we need audio, images, Canvas, WebGL, etc? The web could just be plain text and we’d get most of the “useful” content still, add images and you get a vast majority of it.
But the idea that the web is a rich environment that has all of these bells and whistles is a good thing imo. Yes there’s attack surface to consider, and it’s not negligible. However, the ability to connect so many different things opens up simple access to things that would otherwise require discrete apps and tooling.
One example that kind of blew my mind is that I wanted a controller overlay for my Twitch stream. After a short bit of looking, there isn’t even a plugin needed in OBS (streaming software). Instead, you add a Web View layer and point it to GamePad Viewer[1] and you’re done.
Serial and USB are possibly a boon for very specific users with very specific accessibility needs. Also, iirc some of the early iPhone jailbreaks worked via websites on a desktop with your iPhone plugged into usb. Sure these are niche, and could probably be served just as well or better with native apps, and web also makes the barrier to entry so much lower .
I think the giant major downside, is that they've written a rootkit that runs on everything, and to try to make up for that they want to make it so only sites they allow can run.
It's not really very powerful at all if nobody can use it, at that point you are better off just not bothering with it at all.
The Internet may remain, but the Web may really be dead.
But people do use it, like the both of us right now?
People also use maps, do online banking, play games, start complex interactive learning environments, collaborate in real time on documents etc.
All of that works right now.
What do you mean, you can run whatever you want on localhost, and it's quite easy to host whatever you want for whoever you want too. Maybe the biggest modern added barrier to entry is that having TLS is strongly encouraged/even needed for some things, but this is an easily solved problem.
I don't see how that solves the issue that PSL tries to fix. I was a script kiddy hosting neopets phishing pages on free cpanel servers from <random>.ripway.com back in 2007. Browsers were way less capable then.
It's not even broken as the edge cases are addressed by ad-hoc solutions.
OP is complaining about global infrastructure not having a pristine design. At best it's a complain over a desirable trait. It's hardly a reason to pull the Jr developer card and mindlessly advocate for throwing everything out and starting over.
We live in world where whatever faang adopts is de facto a standard. Accessible these days means google/gmail/facebook/instagram/tiktok works. Everything else is usually forced to follow along.
People will adopt whatever gives them access to their daily dose of doomscrolling and then complain about rather crucial part of their lives like online banking not working.
> And of course, if the new solution completely invalidates old sites, it just won't get picked up.
Old sites don't matter, only high-traffic sites riddled with dark patterns matter. That's the reality, even if it is harsh.
Try 90s! We had to fight off ActiveX Plugins left and right in the good olde Internet Explorer! Yarr! ;-)
This might be what's needed to break out of the current local optimum.
which is still much too new to be able to shut down the PSL of course. but maybe in 2050.
CORS lets sites define their own security boundaries between subdomains, with mutual validation. If you're hosting user content in a subdomain, just don't allow-origin it: that is a clear statement that it's not "the same site". PSL plays absolutely no part in that logic, it seems clear to me that it's at least in part intended to replace the PSL.
Do other sites (like google's safety checks) use CORS for this purpose? Dunno. Seems like they could though? Or am I missing something?
I just came up with that in 2 minutes, so it might not be perfect, but you can see how with a little bit of work there's much better solutions than "I check for not-evil domain in list!"
So now, if a website leaks its private key, attackers can exfiltrate cookies from all of its users just by making them open an attacker-controlled link, for as long as the cookie lives (and users don't visit the website to get the rotated key).
> If the web server needs a cookie, it should request one
This adds a round-trip, which slows down the website on slow connections.
> the client can submit again to "reply" to this "request"
This requires significantly overhauling HTTP and load-balancers. The public-suffix list exists because it's an easy workaround that didn't take a decade to specify and implement.
This attack already exists in several forms (leaking a TLS private key, DNS hijack, CA validation attack, etc). You could tack a DNS name onto the crypto-cookies if you wanted to, but DNS is trivial to attack.
> This adds a round-trip, which slows down the website on slow connections.
Requests are already slowed down by the gigantic amount of cookies constantly being pushed by default. The server can send a reply-header once which will tell the client which URLs need cookies perpetually, and the client can store that and choose whether it sends the cookies repeatedly or just when requested. This gives the client much more control over when it leaks users' data.
> This requires significantly overhauling HTTP and load-balancers
No change is needed. Web applications already do all of this all the time. (example: the Location: header is frequently sent by web apps in response to specific requests, to say nothing of REST and its many different request and return methods/statuses/headers).
> The public-suffix list exists because it's an easy workaround
So the engine of modern commerce is just a collection of easy hacks. Fantastic.
An attacker who gets the TLS private key of a website can't use it easily, because they still need to fool users' browser into connecting to a server they control as the victim domain, which brings us to:
> You could tack a DNS name onto the crypto-cookies if you wanted to, but DNS is trivial to attack.
It's not. I can think of two ways to attack the DNS. Either 1. control or MITM of the victim's authoritative DNS server or 2. poison users' DNS cache.
Control/MITM of the authoritative server is not an option for everyone (only ISPs/backbone operators), and according to Cloudflare: "DNS poisoning attacks are not easy" (https://www.cloudflare.com/learning/dns/dns-cache-poisoning/)
> Requests are already slowed down by the gigantic amount of cookies constantly being pushed by default
Yes, although adding more data and adding a round-trip have different impacts (high-bandwidth high-latency connections exist). Lots of cookies and more round-trips is always worse than lots of cookies and a fewer round-trips.
> The server can send a reply-header once which will tell the client which URLs need cookies perpetually, and the client can store that and choose whether it sends the cookies repeatedly or just when requested.
Everyone hate configuring cache, so in most cases site operators will leave it to a default "send everything", and we're back to square one.
> No change is needed.
I was thinking that servers need to remember state between the initial client request and when the client sends an other request with the cookies. But on second thought that's indeed not necessary.
> So the engine of modern commerce is just a collection of easy hacks. Fantastic.
I'm afraid so
Cache poisoning is the easiest method, and contrary to whatever Cloudflare says, it's trivial. The DNS transaction number is 16-bits. All you have to do is flood the shit out of the resolver with spoofed packets and eventually one of the transaction numbers will hit, and your attack is successful. It's low-bandwidth, takes at most a couple hours, and nobody notices. This is one of the many reasons you can't just trust whatever DNS says.
The choice of what HTTP messages to cache is not always a choice, as is the case with HSTS. But it could be made one if testing of this proposal (which again, I came up with in 2 minutes) showed better results one way or another.
But all this is moot anyway cuz nobody gives a crap.
[0] https://www.soundandvision.com/content/remembering-time-when...
Not really. Does the car still drive? That sounds like a software bug; hardly indicative that the entire car is held together with duct tape, but a pretty bad bug non the less.
It's just harder to drive to one house, and the homeowner is justifiably irritated about this.
Google doesn't have the force of law (it's in this context acting more like a Yelp: "1 star review --- our secret shopper showed up and the manager didn't give the secret 'we are not criminals' hand sign"), but the basic idea is the same: there is a complex web of interactions that can impact your online presence and experts in the field you can choose to hire for consulting or not.
Didn't used to be that way, but the web used to be a community of 100,000 people, not 5.6 billion. Everything gets more complicated when you add more people.
The browser issue can destroy a small business, one thing I think we can universally agree we don't want. If all of the people who come looking for it find it's being marked as malicious or just can't get there at all, they lose customers.
Worse yet, is that Google holds the keys because everyone uses Chrome, and you have to play their game by their rules just to keep breathing.
I honestly don't immediately know how to weigh those risks against each other, but I'll note that this community likely underestimates the second one. Most web users are not nearly as tech- or socially-savvy as the average HN reader and the various methods of getting someone to a malware subdomain are increasingly sophisticated.
"This is Fine."
Before we put computers in cars, we had the myriad small things that would break (stuck doors, stuck windows, failed seals, leaking gaskets), a continuous stream of recalls for low-probability safety issues, and the occasional Gremlin or Pinto.
My favorite example is the Hyundai Elantra. They changed the alloy used in one of the parts in the undercarriage. Tested that model to death for a year, as they do, but their proving ground is in the southern United States.
Several winters later, it turns out that road salt attacks the hell out of that alloy and people have wheels flying off their cars in the middle of the road.
And judging by eBay prices, or the SwissMicros product line, I suspect I have plenty of company.
This is mostly a browser security mistake but also partly a product of ICANN policy & the design of the domain system, so it's not just the web.
Also, the list isn't really that long, compared to, say, certificate transparency logs; now that's a truly mad solution.
A centralized list like this not just for domains as a whole (e.g. co.uk) but also specific sites (e.g. s3-object-lambda.eu-west-1.amazonaws.com) is both kind of crazy in that the list will bloat a lot over the years, as well as a security risk for any platform that needs this functionality but would prefer not to leak any details publicly.
We already have the concept of a .well-known directory that you can use, when talking to a specific site. Similarly, we know how you can nest subdomains, like c.b.a.x, and it's more or less certain that you can't create a subdomain b without the involvement of a, so it should be possible to walk the chain.
Example:
c --> https://b.a.x/.well-known/public-suffix
b --> https://a.x/.well-known/public-suffix
a --> https://x/.well-known/public-suffix
The point is to get away from centralized gatekeepers, not establish more of them. A hierarchy of disavowal. It’s like cache invalidation for accountability.
If you don’t wanna be held responsible for something, you’d better be prepared to point the finger at someone whois.
I’m not sure how you’d have this - it’s for the public facing side of user hosted content, surely that must be public?
> We already have the concept of a .well-known directory that you can use, when talking to a specific site.
But the point is to help identify dangerous sites, by definition you can’t just let the sites mark themselves as trustworthy and rotate around subdomains. If you have an approach that doesn’t have to trust the site, you also don’t need any definition at the top level you could just infer it.
Although technically it might be better as `.wellknown/taint-regex` (now we have three problems), like `TAINT "*.sites.myhost.com" ; "myhost.com/uploads/*" ; ...`
The thing you want to avoid is this:
a.scamsite.com gets blocked so they just put their phishing pages on b.scamsite.com
The psl or your solution isn’t a “don’t trust subdomains” notification it’s “if one subdomain is bad, you should still trust the others” and the problem there is you can’t trust them.
You could combine the two, but you still need the suffix list or similar curation.
There's some ways of like "nofollow", but nothing systematic, and no "protocol" for disavowing paths, uploads, or fragments.
Back in the slashdot days, I thought of "blogs are the stationary of the internet", a way to more authoritatively declare that the content was yours... but interop is hard and unprofitable so walled gardens became the norm.
We just haven't had the benefit or forcing function which encourages a solution to "that stuff over there is less trusted than my stuff over here".
Maybe we're at the point where hosts of any kind MUST be responsible (or accountable) for any content originating from their domain? It kills indie/anonymous hosting, but puts a fine "KYC" point on distributing "evil" stuff on the internet?
> We just haven't had the benefit or forcing function which encourages a solution to "that stuff over there is less trusted than my stuff over here".
No the problem is we can't let people say "that stuff is someone elses fault" when it is their own fault.
Scammers will claim subdomains are actually just not them and are other bad actors, and you're back to loads of phishing pages.
> Maybe we're at the point where hosts of any kind MUST be responsible (or accountable) for any content originating from their domain?
We already are at that point, the PSL is to get past it for cases where people host on subdomains. Netlify shouldn't have to risk having every customer flagged if one customer is a phisher. The curation is vital.
The other solution would be to have another approach around hosting where verifiable owners could publish wherever they want and it's tied to a real entity, but that has other worrying outcomes I assume.
The other way of looking at it might be similar to "DMARC-4-HTTP", ie: sign Content-Length, Content-Sig with a public/private key and if you include `SELECT comments FROM evil` then that "taints" your key.
It gets back to netlify that index.html would be signed by netlify.gpg, but haxor.netlify.com would be signed by not_netlify.gpg
...we can call it "web of trust 2.0" :-P
Appreciate the honest discussion!
One key difference between this list and standard DNS (at least as I understand it; maybe they added an extension to DNS I haven't seen) is the list requires independent attestation. You can't trust `foo.com` to just list its subdomains; that would be a trivial attack vector for a malware distributor to say "Oh hey, yeah, trustme.com is a public suffix; you shouldn't treat its subdomains as the same thing" and then spin up malware1.trustme.com, malware2.trustme.com, etc. Domain owners can't be the sole arbiter of whether their domain counts as a "public suffix" from the point of view of user safety.
$ dig +short txt _psl.website.one @1.1.1.1
"https://github.com/publicsuffix/list/pull/2625"
Most of the complex thing I have seen being made (or contributed to) needed duct tape sooner or later. Engineering is the art of trade-offs, of adapting to changing requirements (that can appear due to uncontrollable events external to the project), technology and costs.
Related, this is how the first long distance automobile trip was done: https://en.wikipedia.org/wiki/Bertha_Benz#First_cross-countr... . Seems to me it had quite some duct tape.
Web is not a new thing and hardly a technical experiment of a few people any more.
If you add the time since announcing the concept of Web to that trip date, you have a very decent established industry already. With many sport and mass production designs:
https://en.wikipedia.org/wiki/Category:Cars_introduced_in_19...
Not sure what is your point about "decent established industry" if we relate to "duct tape". I see two possibilities:
a) you imply that the web does not have a decent established industry (but I would guess not).
b) you would claim that there was no "duct tape" in 1924 car industry. I am no expert but I would refer you to the article describing what was the procedure to start the car at https://www.quora.com/How-do-people-start-their-cars-in-the-..., to quote:
> Typical cold-start routine (common 1930s workflow)
> 1. Set hand choke (pull knob).
> 2. Set throttle lever to slight fast‑idle.
> 3. Retard spark if manual advance present.
> 4. Engage starter (electric) or use hand crank.
> 5. Once running, push choke in gradually, advance spark, reduce throttle.
Not sure about your opinion but compared to what a car's objective is (move from point A to point B) to me that sounds rather involved. Not sure if it qualifies as "duct-tape" but definitely it is not a "nicely implemented system that just works".
To resume my point: I think on average progress is slower and harder than people think. And that is mostly because people do not have exposure to the work people are doing to improve things until something can become more "widely available".
Idk any other way to solve it for the general public (ideally each user would probably pick what root certs they trust), but it does seem crazy.
This solution is even more obvious today where most certificates are just DNS lookups with extra steps.
1. Immich hosts user content on their domain. And should thus be on the public suffic list.
2. When users host an open source self hosted project like immich, jellyfin, etc. on their own domain it gets flagged as phishing because it looks an awful lot like the publicly hosted version, but it's on a different domain, and possibly a domain that might look suspicious to someone unfamiliar with the project, because it includes the name of the software in the domain. Something like immich.example.com.
The first one is fairly straightforward to deal with, if you know about the public suffix list. I don't know of a good solution for the second though.
I get that SPAM, etc., are an issue, but, like f* google-chrome, I want to browse the web, not some carefully curated list of sites some giant tech company has chosen.
A) you shouldn't be using google-chrome at all B) Firefox should definitely not be using that list either C) if you are going to have a "safe sites" list, that should definitely be a non-profit running that, not an automated robot working for a large probably-evil company...
It's browser beware when you do, but you can do it.
The problem is that at least some of the people maintaining this list seem to be a little trigger happy. And I definitely thing Google probably isn't the best custodian of such a list, as they have obvious conflicts of interest.
And how and who should define what is consider unsafe sites?
But this is not that list because sites are added using opaque automated processes that are clearly not being reviewed by humans - even if those sites have been removed previously after manual review.
People are reacting as if this list is some kind of overbearing way of tracking what people do on the web - it's almost the opposite of that. It's worth clarifying this is just a suffix list for user-hosted content. It's neither a list of user-hosted domains nor a list of safe websites generally - it's just suffixes for a very small specific use-case: a company providing subdomains. You can think of this as a registry of domain sub-letters.
For instance:
- GitHub.io is on the list but GitHub.com is not - GitHub.com is still considered safe
- I self-host an immich instance on my own domain name - my immich instance isn't flagged & I don't need to add anything to the list because I fully own the domain.
The specific instance is just for Immich themselves who fully own "immich.cloud" but sublet subdomains under it to users.
> *if you are going to have a "safe sites" list"
This is not a safe sites list! This is not even a sites list at all - suffixes are not sites. This also isn't even a "safe" list - in fact it's really a "dangerous" list for browsers & various tooling to effectively segregate security & privacy contexts.
Google is flagging the Immich domain not because it's missing from the safe list but because it has legitimate dangers & it's missing from the dangerous list that informs web clients of said dangers so they can handle them appropriately.
I've coined the phrase "Postel decentralization" to refer to things where people expect there to be some distributed consensus mechanism but it turned out that the design of the internet was to email Jon Postel (https://en.wikipedia.org/wiki/Jon_Postel) to get your name on a list. e.g. how IANA was originally created.
I was just deploying your_spotify and gave it your-spotify.<my services domain> and there was a warning in the logs that talked about thud, linking the issue:
I know the second issue can be a legitimate problem but I feel like the first issue is the primary problem here & the "solution" to the second issue is a remedy that's worse than the disease.
The public suffix list is a great system (despite getting serious backlash here in HN comments, mainly from people who have jumped to wildly exaggerated conclusions about what it is). Beyond that though, flagging domains for phishing for having duplicate content smells like an anti-self-host policy: sure there's phishers making clone sites, but the vast majority of sites flagged are going to be legit unless you employ a more targeted heuristic, but doing so isn't incentivised by Google's (or most company's) business model.
The fact it's used by one or more browsers in that way is a lawsuit waiting to happen.
Because they, the browsers, are pointing a finger to someone else and accusing them of criminal behavior. That is what a normal user understands this warning as.
Turns out they are wrong. And in being wrong they may well have harmed the party they pointed at, in reputation and / or sales.
It's remarkable how short sighted this is, given that the web is so international. Its not a defense to say some third party has a list, and you're not on it so you're dangerous
Incredible
You're honor, we hurt the plaintiff because it's better than nothing!
The point I raise is that the internet is international. There are N legal systems that are going to deal with this. And in 99% of them this isn't going to end well for Google if plaintiff can show there are damages to a reasonable degree.
It's bonkers in terms of risk management.
If you want to make this a workable system you have to make it very clear this isn't necessarily dangerous at all, or criminal. And that a third party list was used, in part, to flag it. And even then you're impeding visitors to a website with warnings without any evidence that there is in fact something wrong.
If this happens to a political party hosting blogs, it's hunting season.
Lacking a global authority, Google is right to implement a filter themselves. Most people are really really dumb online and if not as clearly "DO NOT ENTER" as now, I don't think the warnings will work. I agree that from a legal standpoint it's super dangerous. Content moderation (which is basically what this is) is an insanely difficult problem for any platform.
I appreciate the issue it tries to solve but it doesn't seem like a sane solution to me.
Browsers already do various levels of isolation based on domain / subdomains (e.g. cookies). PSL tells them to treat each subdomain as if it were a top level domain because they are operated (leased out to) different individuals / entities. WRT to blocking, it just means that if one subdomain is marked bad, it's less likely to contaminate the rest of the domain since they know it's operated by different people.
Informing decisions about blocking doesn't make much sense (IMO) because it's little more than a speed bump for an attacker. Certainly every little bit can potentially help but it also introduces a new central authority, presents an additional hurdle for legitimate operators, introduces a number of new failure modes, and in this case seems relatively trivial for a determined attacker to overcome.
What would you recommend for this actual use case? Even splitting it off to a separate domain name as they’re planning merely reduces the blast radius of Google’s false positive, but does not eliminate it.
None of these are the kind of automatic user-generated content that the warning is attempting to detect, I think. And requiring basic auth for everything is quite awkward, especially if the deployment includes API server functionality with bearer token auth combined with unauthenticated endpoints for things like built-in documentation.
This also polluted their own domain, even when the redirect was removed, and had the odd side effect that Google would no longer accept email from them. We requested a review and passed it, but the email blacklist appears to be permanent. (I already checked and there are no spam problems with the domain.)
We registered a new domain. Google’s behaviour here incidentally just incentivises bulk registering throwaway domains, which doesn’t make anything any better.
And avoid using subdomains.
0: https://old.reddit.com/r/immich/comments/1oby8fq/immich_is_a...
https://blog.chromium.org/2021/07/m92-faster-and-more-effici...
Not sure if this is exactly the scenario from the discussed article but it's interesting to understand it nonetheless.
TL;DR the browser regularly downloads a dump of color profile fingerprints of known bad websites. Then when you load whatever website, it calculates the color profile fingerprint of it as well, and looks for matches.
(This could be outdated and there are probably many other signals.)
https://github.com/immich-app/devtools/tree/a9257b33b5fb2d30...
These days, when I get a capitalized, grammatically correct sentence — and proper punctuation to boot, there is an unfortunate chance it was written using an AI and I am not engaging fully with a human.
its when my covnersation partner makes human mistakes, like not capitalizing things, or when they tell me i'm a bonehead, that i know i'm talking to a real human not a bot. it makes me feel happier and more respected. i want to interact with humans dammit, and at this point rude people are more likely to be human than polite ones on the internet.
i know you can prompt AIs to make releaistic mistakes too, the arms race truly never ends
I seem to have irritated the parallel commenters tremendously by asking, but it seemed implausible I’d understand the design considerations by just skimming the CI config.
Top of mind would be:
1. How do y'all think about mitigating the risk of somebody launching malicious or spammy PR sites? Is there a limiting factor on whose PRs trigger a launch?
2. Have you seen resource constraint issues or impact to how PRs are used by devs? It seems like Immich is popular enough that it could easily have a ton of inflight PR dev (and thus a ton of parallel PR instances eating resources)
3. Did you borrow this pattern from elsewhere / do you think the current implementation of CI hooks into k8s would be generalizable? I’ve seen this kind of PR preview functionality in other repos that build assets (like CLI tools) or static content (like docs sites), but I think this is the first time I’ve seen it for something that’s a networked service.
2. It's running on a pretty big machine, so I haven't seen it approach any limits yet. We also only create an instance when requested (with a PR label).
3. I've of course been inspired by other examples, but I think the current pattern is mostly my own, if largely just one of the core uses of the flux-operator ResourceSet APIs [1]. It's absolutely generalizable - the main 'loop' [2] just templates whatever Kubernetes resources based on the existence of a PR, you could put absolutely anything in there.
[1]: https://fluxcd.control-plane.io/operator/resourcesets/github...
[2]: https://github.com/immich-app/devtools/blob/main/kubernetes/...
Things cost money, and at a large scale, there's either capitalism, or communism.
Can you point them out on the map?
Google's eternal vagueness is infuriating but in this case the whole setup is a disaster waiting to happen. Google's accidental fuck-up just prevented "someone hacked my server after I clicked on pr-xxxx.imiche.app" because apparently the domain's security was set up to allow for that.
You can turn off safe browsing if you don't want these warnings. Google will only stop you from visiting sites if you keep the "allow Google to stop me from visiting some sites" checkbox enabled.
They used to be more generic saying "We don't know if its safe" but now they are quite assertive at stating you are indeed an attacker.
I’m not a lawyer, but this hasn’t ever been taken to court, has it? It might qualify as libel.
Our lawyers spoke to Google's lawyers privately, and our domains got added to a whitelist at Google.
Likewise, if it's a fuckup that just needs to be put in front of someone who cares, a lawsuit is actually a surprisingly effective way of doing that. This moves your problem from "annoying customer support interaction that's best dealt with by stonewalling" into "legal says we HAVE to fix this".
No they're not. The word "scammer" does not appear. They're saying attackers on the site and they use the word "might".
This includes third-party hackers who have compromised the site.
They never say the owner of the site is the attacker.
I'm quite sure their lawyers have vetted the language very carefully.
I think that might count as libel.
Yes, the question was literally about the law.
I wasn't trying to say anything else. I was answering the commenter's legal question.
I don't know what you are trying to imply.
If the false positive rate is consistently 0.0%, that is a surefire sign that the detector is not effective enough to be useful.
If a false positive is libel, then any useful malware detector would occasionally do libel. Since libel carries enormous financial consequences, nobody would make a useful malware detector.
I am skeptical that changing the wording in the warning resolves the fundamental tension here. Suppose we tone it down: "This executable has traits similar to known malware." "This website might be operated by attackers."
Would companies affected by these labels be satisfied by this verbiage? How do we balance this against users' likelihood of ignoring the warning in the face of real malware?
They could at least send a warning email to the RFC2142 abuse@ or hostmaster@ address with a warning and some instructions on a process for having the mistake reviewed.
Now imagine it goes one step further, and when you go to eat the food anyway, your Walmart fork retracts into its handle for your safety, of course.
No brand or food supplier would put up with it.
That's what it's like trying to visit or run non-blessed websites and software coming from Google, Microsoft, etc on your own hardware that you "own".
Being wrong doesn't count as libel.
If a company has a detection tool, makes reasonable efforts to make sure it is accurate, and isn't being malicious, you'll have a hard time making a libel case
For instance: https://reason.com/volokh/2020/07/27/injunction-in-libel-cas... (That was a default judgment, though, which means Spamhaus didn't show up, probably due to jurisdictional questions.)
The first step in filing a libel lawsuit is demanding a retraction from the publisher. I would imagine Google's lawyers respond pretty quickly to those, which is why SafeBrowsing hasn't been similarly challenged.
This is a spicy one, would love to know more.
I know that I can bypass the warning, but the photo album I sent to my mother-in-law is now effectively inaccessible.
That area will show you the exact URLs that got you put on the block list. You can request a review from there. They'll send you an email after they review the block.
Hopefully that'll save you from trying to hunt down non-existent malware on a half dozen self-hosted services like I ended up doing.
I guess a workaround Google's crap would be to put an htpasswd/basic auth in front of Immich, blocking Google to get to the content and flagging it.
Btw, folks in the Jellyfin thread tried blocking specifically Google bot / IP ranges (ASNs?) https://github.com/jellyfin/jellyfin-web/issues/4076#issueco... with varying success.
And go through your domain registration/re-review in G Search Console of course.
Thus if I started hosting my Immich instance, I would probably put it behind "pxl.domain.tld" or something like that.
Not a garantee to pass the Google purity test, but, according to some reports, it would avoid raising some redflags.
It may well be a false positive of Google's heuristics but home server security can be challenging - I would look at ruling out the possibility of it being real first.
It certainly sounds like a separate root issue to this article, even if the end result looks the same.
https://news.ycombinator.com/item?id=42779544#42783321
Unironically, including a threat of legal action in my appeal on the Google Search Console was what stopped our instance getting flagged in the end.
Is a notion page, github repo, or google doc that has user submitted content that can be publicly shared also user-hosted?
IMO Google should not be able to use definitive language "Dangerous website" if its automated process is not definitive/accurate. A false flag can erode customer trust.
The definition of "active code" is broad & sometimes debatable - e.g. do old MySpace websites count - but broadly speaking the best way of thinking about it is in terms of threat model, & the main two there are:
- credential leakage
- phishing
The first is fairly narrow & pertains to uploading server side code or client javascript. If Alice hosts a login page on alice.immich.cloud that contains some session handling bugs in her code, Mallory can add some cute to mallory.immich.cloud to read cookies set on *.immich.cloud to compromise Alice's logins.
The second is much broader as it's mostly about plausible visual impersonation so will also cases where users can only upload CSS or HTML.
Specifically in this case what Immich is doing here is extremely dangerous & this post from them - while I'll give them the benefit of the doubt on being ignorant - is misinformation.
By preventing newcomers from using this pattern, Google's system is flawed, severely stifling competition.
Of course, this is perfectly fine for Google.
1. Use a separate dedicated domain (Immich didn't do this - they're now switching to one in response to this)
2. List the separate dedicated domain in the public suffix list. As far as I can tell Immich haven't mentioned this.
You fully misunderstand what content is hosted on these sites. It's only builds from internal branches by the core team, there is no path for "external user" content to land on this domain.
Correct. It works this way because in general the domain has the rights over routing all the subdomains. Which means if you were a spammer, and doing something untoward on a subdomain only invalidated the subdomain, it would be the easiest game in the world to play.
malware1.malicious.com
malware2.malicious.com
... Etc.
So is there someone from Google around who can send this along to the right team to ensure whatever heuristic has gone wrong here is fixed for good?
The best we all can do is to stop using Google products and encourage our friends and family to do likewise. Make sure in our own work that we don't force others to rely on Google either.
I had prior been tossing up the pros/cons of this (such as teaching the user to accept millions of arbitrary TLDs as official), but I think this article (and other considerations) have solidified it for me.
For example
www.contoso.com (public)
www.contoso.blog (public with user comments)
contoso.net (internal)
staging.contoso.dev (dev/zero trust endpoints)
raging-lemur-a012afb4.contoso.build (snapshots)
It happened to me a while ago that I suddenly got emails from "githubnext.com". Well, I know Github and I know that it's hosted at "github.com". So, to me, that was quite obviously phishing/spam.
Turns out it was real...
And new TLDs are coming out every day which means that I could probably go buy microsoft.anime if I wanted it.
This is what trademarks are supposed to do, but it’s reactive and not proactive.
Real users don't care much about phishing as long as you got redirected from the main domain, though. github.io has been accepted for a long time, and githubusercontent.com is invisible 99% of the time. Plus, if your regular users are not developers and still end up on your dev/staging domains, they're bound to be confused regardless.
Something Google actively facilities with the ads they serve.
We have an iOS app in the store for 3 years and out of the blue apple is demanding we provide new licenses that don’t exist and threaten to kick our app out. Nothing changed in 3 years.
Getting sick of these companies able to have this level of control over everything, you can’t even self host anymore apparently.
Crazy! If you can elaborate here, please do.
Surely among us devs, as we realize app stores increasingly hostile, that the open web is worth fighting for, and that we have the numbers to build solutions?
[Sam didn't like that.]
Because they're so reliable on Google funding, they're trying to do whatever they can to find alternative revenue streams. Damned if you do, damned if you don't, especially for the HN crowd.
I don't even have a Home button that I can see, I must have turned it off in settings? I describe my tab count using scientific notation, though, so I'd be a "new tab" guy, anyway. But I'd also be a proponent of it being configurable.
I'm not sure why his reaction would be relevant, though. It'll just be another rant about how Google has too much control like he's done in the past. He may be right, but there's nothing new to say.
> Wouldn't be surprised if this leads to a lawsuit over monopolistic behavior
His reaction also matters because he's basically the public face for the company on YouTube and has a huge following. You've probably seen a bunch of social media accounts with the "clippy" character as their avatar. That's a movement started by Louis Rossman.
They did a similar thing with the uBlock Origin extension, flagging it with “this extension might be slowing down your browser” in a big red banner in the last few months of manifest v2 on Chrome. After already having to upload the extension yourself to Chrome cause they took it off the extension store cause it was inhibiting on their ad business.
Google is a massive monopolistic company who will pull strings on one side of their business to help another.
With only Firefox not being based on Chromium and still having manifest v2 the future (5 to 10 years from now) looks bleak. With only 1 browser like this web devs can phase it out slowly by not taking it into consideration when coding or Firefox could enshittify to such an extent because of their manifest v2 monopoly that even that wont make it worth it anymore.
Oh and for the ones not in the know, Manifest is the name of a javascript file manifest.js that decides what browser extensions can and cant modify and the “upgrade” from manifest v2 to v3 has made it near impossible for adblockers to block ads.
But how effective is it in malware detection?
The benefits seem to me dubious. It looks like a feature offered to collect browsing data, useful to maybe 1% in special situations.
The address appears to be adsense.google.com.
Reminds me of MS blocking a website of mine for dangerous script. The offending thing i did was use document.write to put copyright 2025 (with the current year) at the end of static pages.
The integrated button to join a Microsoft Teams meeting directly from my Microsoft Outlook Calendar doesn't work because Microsoft needs to scan the link from Microsoft to Microsoft for malware before proceeding, and the malware scanning service has temporary downtime and serves me static page saying "The content you are accessing cannot currently be verified".
Enshitification ensues.
They just need to introduce a basic deposit to post ads, and you lose it if you put up a scam ad. Would soon pay for the staff needed to police it, and prevent scammers from bypassing admin by trivially creating new accounts.
The reality is that google profits from scam adverts, so they don't proactively do anything about it and hide behind the "at our scale, we can't effectively do anything about it" argument. Which is complete horseshit because if you can't prevent obvious scams on your platform, you don't deserve to have a platform. Google doesn't have to be running at their scale. "We would make less money" is not a valid excuse. We'd all make more money if we could ignore laws and let people be scammed or taken advantage of.
There's plenty of ways they could solve it, but they choose not to. IMHO this should be a criminal offence and google executives should be harshly punished. Its also why I have a rather negative view of googlers, since they wilfully perpetuate this stuff by working on adtech while nothing is being done about the normal everyday people getting scammed each day. Its only getting worse with AI, but I've been seeing it for years.
no longer able? or no longer willing to, because it impacts their bottom line?
I was able to block most of this via ublock origin but Google disabled this - can not download it from here anymore:
https://chromewebstore.google.com/detail/ublock-origin/cjpal...
Funniest nonsense "explanation":
"This extension is no longer available because it doesn't follow best practices for Chrome extensions."
In reality Google killed it because it threatens their greed income. Ads, ads and more ads.
Librewolf is just a directly de-mozillaed and privacy-enhanced Firefox, similar to Ungoogled Chromium. I've been trying to get in the habit of using Zen Browser, which has a bunch of UI changes.
Those terms of use aren't in place any longer. I'm surprised that listening to the users is viewed as something bad.
Their marketing and legal departments lost it long before the terms of service debacle.
edit: also, by "libweb", did you mean "ladybird"?
Typical freeloader behaviour, moans about free software politics but won't contribute anything themselves.
Librewolf is trying to be de-Mozillaed, privacy-enhanced Firefox, so it'll probably take whatever not-overtly-spyware patches Mozilla adds. Some others, like Waterfox and Pale Moon, are more selective.
If we had functional anti-trust laws then this company would have been broken up long ago, Alphabet or not. But they keep doing these things because we - collectively - let them.
Reading a bit online (not having any personal/deep knowledge) it seems the original extension also downloaded updates from a private (the developers) server, while that is no longer allowed - they now need to update via the chrome extension, which also means waiting for code review/approval from google.
I can see the security angle there, it is just awkward how much of an vested interest google has in the whole topic. ad-blocking is already a grey area (legally), and there is a cat-and-mouse between blockers and advertisers; it's hard to believe there is only security best-practise going on here.
I'm still amazed how deploying spyware would've rightfully landed you in jail a couple decades back, but do the same thing on the web under the justification of advertising/marketing and suddenly it's ok.
I'm pretty sure that if Springer were to make a fraudulent ad, they would instantly be slapped with a lawsuit and face public outcry.
Google and Meta are trillion dollar criminal enterprises. The lion's share of their income comes from fraud and scams, with real victims having their lives destroyed. That is the sad truth, no matter how good and important some of their services are. They will never stop their principal source of income.
I actually find the semantic search of immich slightly better.
Essentially CLIP lets to encode both text and images in same vector space.
It is really easy and pretty fast too generate embeddings. Took less than hour on Google Colab.
I made a quick and dirty Flask app that lets me query my own collection of pictures and provide most relevant ones via cosine similarity.
You can query pretty much anything on CLIP (metaphors, lightning, object, time, location etc).
From what I understand many photo apps offer CLIP embedding search these days including Immich - https://meichthys.github.io/foss_photo_libraries/
Alternatives could be something like BLIP.
ViT-SO400M-16-SigLIP2-384__webli
I think I found it because it was recommended by Immich as the best, but it still only took a day or two to run against my 5 thousand assets. I’ve tested it against whatever Google is using (I keep a part of my library on Google Photos), and it’s far better.
When I worked customer service, these phishing blocks worked wonders preventing people from logging in to your-secure-webmail.jobz. People would be filling in phishing forms days after sending out warnings on all official channels. Once Google's algorithm kicked in, the attackers finally needed to switch domains and re-do their phishing attempts.
It's also in general a thankless job maintaining any open-source project, especially one of this scale, so a certain level of kneejerk cynical dismissiveness around stuff like this is expected & very forgivable.
Just really hope the ignorance / knowledge-gap can be closed off though, & perhaps some corrections to certain statements published eventually.
1. You should host dev stuff and separate domains.
2. Google shouldn't be blocking your preview environments.
Is it far fetched that the people controlling a subdomain may not be the same that control the domain?
To be clear, the issue here is that some subdomains pose a risk to the overall domain - visiting any increases your risk from others. It's also related to a GitHub workflow that auto-generates new subdomains on demand, so there's no possibility to have a fixed list of known subdomains since new ones are constantly being created.
Similarly, if an organization deploys a public system that engages in libel and tortious interference, the fact that jumping through technical hoops might make it less likely to be affected by that system does not in any way absolve the organization for operating it carelessly in the first place.
Just because there are steps you can take to lessen the impact of bad behavior does not mean that the behavior itself isn't bad. You shouldn't have restrict how you use your own domains to avoid someone else publishing false information about your site. Google should be responsible for mitigating false positives, not the website owners affected by them.
First & foremost I really need to emphasise that, despite the misleading article title, this was not a false positive. Google flagged this domain for legitimate reasons.
I think there's likely a conversation to be had about messaging - Chrome's warning page seems a little scarier than it should be, Firefox's is more measured in its messaging. But in terms of the API service Google are providing here this is absolutely not a false positive.
The rest of your comment seems to be an analoy about people not being responsible for protecting their home or something, I'm not quite sure. If you leave your apartment unlocked when you go out & a thief steals your housemate's laptop, is your housemate required to exclusively focus on the thief or should they be permitted to request you to be more diligent about locking doors?
Why would it flag a domain rather than a subdomain?
No they didn't.
Where are you getting that from? I don't see any evidence that there actually was any malicious activity going on on the Immich domain.
> But in terms of the API service Google are providing here this is absolutely not a false positive.
Google is applying heuristics derived from statistical correlations to classify sites. When a statistical indicator is present, but its target variable is not present, that is the very definition of a false positive.
Just because their verbiage uses uncertainty qualifiers like "may" or "might" doesn't change the fact that they are materially interfering with a third party's activities based on presumptive inferences that have not been validated -- and in fact seem to be invalid -- in this particular case.
> If you leave your apartment unlocked when you go out & a thief steals your housemate's laptop, is your housemate required to exclusively focus on the thief or should they be permitted to request you to be more diligent about locking doors?
One has nothing to do with the other. The fact that you didn't lock your door does not legitimize the thief's behavior. Google's behavior is still improper here, even if website operators have the option of investing additional time, effort, or money to reduce the likelihood of being misclassified by Google.
The target variable is user hosted content on subdomains of a domain not listed in Mozilla's public suffix list. Firefox & Chrome apply a much stricter set of security settings for domains on that list, due to the inherent dangers of multiuser domains. That variable is present, Immich have acknowledged it & are migrating to a new domain (which they will hopefully add to Mozilla's list).
> The fact that you didn't lock your door does not legitimize the thief's behavior. Google's behavior is still improper here
I made no claims about legitimising the thief's behaviour - only that leaving your door unlocked was negligent from the perspective of your housemate. That doesn't absolve the thief. Just as any malicious actor trying to compromise Immich users would still be the primary offender here, but that doesn't absolve Immich of a responsibility to take application security seriously.
And I don't really understand where Google fits in your analogy? Is Google the thief? It seems like a confusing analogy.
No, that's the indicator. The target variable is "malicious website".
Judging by what a person from the Immich team said, that does not seem to be true?
> the whole system only works for PRs from internal branches - https://news.ycombinator.com/item?id=45681230
So unless one of the developers in the team published something malicious through that system, it seems Google did not have a legitimate reason for flagging it.
If that happened we'd have much bigger problems than Google's flagging.
Because the article seems to only ever get an excuse from Google that is easy to dismiss because most sites do something similar.
Domains with user generated active content should typically by listed on Mozilla's Public Suffix list, which Firefox & Chrome both check & automatically apply stricter security settings to, to protect users.
No it's not
> PRs can be autodeployed to this domain without passing review or approval.
No they can't
There is no untrusted/user content on these domains.
In fairness, in my local testing sofar, it appears to be an entirely unauthenticated/credential-less service so there's no risk to sessions right now for this particular use-case. That leaves the only risk-factors being phishing & deploy environment credentials.
During the appeal it was reviewed from India, and I had been using geoblocking. This caused my appeal to be denied.
I ended up deploying to a new domain and starting over.
Never caught back up.
I have been ashamed of it at multiple times in my career, but tbh I built something that fed, sheltered, and clothed me. It was worth.
I also learned a lot.
Makes precisely zero sense.
The nerdsphere has been buzzing with Immich for some time now (I started using it a month back and it lives up to its reputation!), and I assume a lot of Googlers are in that sphere (but not neccessarily pro-Google/anti-Immich of course). So I bet they at least know of it. But do they talk about it?
Google is an evil company I want the web to be free of, I resent that even Firefox & Safari use this safe browsing service. Immich is a phenomenal piece of software - I've hosted it myself & sung its praises on HN in the past.
Put putting aside David vs Goliath biases here, Google is 100% correct here & what Immich are doing is extremely dangerous. The fact they don't acknowledge that in the blog post shows a security knowledge gap that I'm really hoping is closed over the course of remediating this.
I don't think the Immich team mean any harm but as it currently stands the OP constitutes misinformation.
I've read the article and don't see anything dangerous, much less extremely so. Care to explain?
1. PR deploys on public repos are inherently tricky as code gains access to the server environment, so you need to be diligent about segregating secrets for pr deployments from production secret management. That diligence is a complex & continuous undertaking, especially for an open source project.
2. Anyone with a GitHub account can use your domain for phishing scams or impersonation.
The second issue is why they're flagged by Google (he first issue may be higher risk to the Immich project but it's out of scope for Google's safe browsing service).
To be clear: this isn't about people running their own immich instance. This is about members of the public having the ability to deploy arbitrary code without review.
---
The article from the Immich team does mention they're switching to using a non-production domain (immich.build) for their PR builds which does indicate to me they somewhat understand the issue (though they've explained it badly in the article), but they don't seem to understand the significance or scope.
This part is not correct: the "preview" label can be set only by collaborators.
> a subdomain of a domain that they also use for production traffic
To clarify this part: the only production traffic that immich.cloud serves are static map tiles (tiles.immich.cloud)
Overall, I share your concerns, and as you already mentioned, a dedicated "immich.build" domain is the way to go.
That's good & is a decent starting point. A decent second step might be to have the Github Actions workflow also check the approval status of the PR before deploying (requiring all collaborators to be constantly aware that the risk of applying a label is similar to that of an approval seems less viable)
Not advice with some time to fix any possible problem, just blocked.
We gave very bad image to our clients and users, and had to give explanations of a false positive from google detection.
The culprit, according to google search console, was a double redirect on our web email domain (/ -> inbox -> login).
After just moving the webmail to another domain, removing one of the redirections just in case, and asking politely 4 times to be unblocked.. took about 12 hours. And no real recourse, feedback or anything about when its gonna be solved. And no responsibility.
The worse is the feeling of not in control of your own business, and depending on a third party which is not related at all with us, which made a huge mistake, to let out clients use our platform.
We need something that's good for small and medium businesses again, local news and get an actual marketplace going - you know what the internet actually promised.
Anyone working on something like this?
What we have is the best sim env to see how stuff shape up. So fixing it should be the aim, avoiding will get us on similar spirals. We'll just go on circles.
But the Stupid Money Govporation must be dethroned first, and I honestly don't see how that could happen without the help of an ELE like a good asteroid impact.
They are currently experimenting with replicating many types of services which are currently websites as protocols with data types, with the goal being that all of these services can share available data with eachother openly.
It's definitely more of a "bazaar" model over a "catherdral" model, with many open questions and it's also tough to get a good overview of what is really going on there. But at least it's an attempt.
Companies have economy of scale (Google, for instance, is running dozens to hundreds of web apps off of one well-maintained fabric) and the ability to force consolidation of labor behind a few ideas by controlling salaries so that the technically hard, detailed, or boring problems actually get solved. Open source volunteer projects rarely have either of those benefits.
In theory, you could compete with Google via
- Well-defined protocols
- That a handful of projects implement (because if it's too many, you split the available talent pool and end up with e.g. seven mediocre photo storage apps that are thin wrappers around a folder instead of one Google Photos with AI image search capability).
- Which solve very technically hard, detailed, or boring technical problems (AI image search is an actual game-changer feature; the difference between "Where is that one photo I took of my dog? I think it was Christmas. Which Christmas, hell I don't know" and "Show me every photo of my dog, no not that dog, the other dog").
I'd even risk putting up bullet point four: "And be willing to provide solutions for problems other people don't want solved without those other people working to torpedo your volunteer project" (there are lots of folks who think AI image detection is de-facto evil and nobody should be working on it, and any open source photo app they can control the fate of will fall short of Google's offering for end-users).
Any new protocol not only has to overcome the huge incumbent that is the web, it has to do so grassroots against the power of global capital (trillions of dollars of it). Of course, it also has to work in the first place and not be captured and centralised like another certain open and decentralised protocol has (i.e., the Web).
Is that easier than the states doing their jobs and writing a couple pages of text?
A structural solution is to educate and lift the whole population to better understand the implications of their choices.
A tactic solution is to try to limit the collusion of decision people and private entities, but this does not seem to go extremely well.
An "evolutionary" solution (that just happens) used to be to have a war - that would push a lot of people to look for efficiency rather than for some interests. But this is made more complex by nukes.
The problem is that the divide of alignment of interests there is between new, small companies and users. New companies want to put up a website without tripping over one of the thousand unwritten rules of "How to not look like a phishing site or malware depot" (many of which are unwritten because protecting users and exploiting users is a cat-and-mouse game)... And users don't want to get owned.
Shard Chrome off from Google and it still has incentives to protect users at the cost of new companies' ease of joining the global network as a peer citizen. It may have less signal as a result of a curtailed visibility on the state of millions of pages, but the consequence of that is that it would offer worse safe browsing protection and more users would get owned as a result.
Perhaps the real issue is that (not unlike email) joining the web as a peer citizen has just plain gotten harder in the era of bad actors exploiting the infrastructure to cause harm to people.
Like... You know what never has these problems? My blog. It's a static-site-generated collection of plain HTML that updates once in a blue moon via scp. I'm not worried about Google's safe browsing infrastructure, because I never look like a malware site. And if I did trip over one of the unwritten rules (or if attackers figured out how to weaponize something personal-blog-shaped)? The needs of the many justify Chrome warning people before going to my now-shady site.
Some candidate language:
- Monopolistic companies may not actively impose restrictions which harm others (includes businesses)
or
- Some restrictions are allowed, but the company must respond to an appeal of restrictions within X minutes; Appeals to the company can themselves be appealed to a governmental independent board which binds the company with no further review permitted; All delays and unreasonable responses incur punitive penalties as judged by the board; All penalties must be paid immediately
or
- If an action taken unilaterally by a company 1) harms someone AND 2) is automated: Then, that automation must be immediately, totally, and unconditionally reversed upon the unilateral request of the victim. The company may reinstate the action upon the sworn statement of an employee that they have made the decision as a human, and agree to be accountable for the decision. The decision must then follow the above appeals process.
or
- No monopolies allowed
That's not generally how monopoly is interpreted in the US (although jurisprudence on this may be shifting). In general, the litmus test is consumer harm. A company is allowed to control 99% of the market if they do it by providing a better experience to consumers than other companies can; that's just "being successful." Microsoft ran afoul of antitrust because their browser sucked and embedding it in the OS made the OS suck too; if they hadn't tried to parlay one product into the other they would be unlikely to have run afoul of US antitrust law, and they haven't run afoul of it over the fact that 70-90% of x86 architecture PCs run Windows.
> Some restrictions are allowed, but the company must respond to an appeal of restrictions within X minutes; Appeals to the company can themselves be appealed to a governmental independent board which binds the company with no further review permitted; All delays and unreasonable responses incur punitive penalties as judged by the board; All penalties must be paid immediately
There may be meat on those bones (a general law restricting how browsers may operate in terms of rendering user content). Risky because it would codify into law a lot of ideas that are merely technical specifications (you can look to other industries to see the consequences of that, like how "five-over-ones" are cropping up in cities all over the US because they satisfy a pretty uniform fire and structural safety building code to the letter). But this could be done without invoking monopoly protection.
> If an action taken unilaterally by a company 1) harms someone AND 2) is automated: Then, that automation must be immediately, totally, and unconditionally reversed upon the unilateral request of the victim.
Too broad. It harms me when Google blocks my malware distribution service because I'm interested in getting malware on your machine; I really want your Bitcoin wallet passwords, you see. ;)
Most importantly: this whole topic is independent of monopolies. We could cut Chrome out of Google tomorrow and the exact same issues with safe browsing impeding new sites with malware-ish shapes would exist (with the only change probably being the false positive rate would go up, since a Chrome cut off from Google would have to build out its detection and reporting logic from scratch without relying on the search crawler DB). More importantly, a user can install another browser that doesn't have site protection today (or, if I understand correctly, switch it off). The reason this is an issue is that users like Chrome and are free to use it and tend to find site protection useful (or at least "not a burden to them") and that's not something Google imposed on the industry, it's a consequence of free user choice.
That's okay, a random company failing to protect users from harm is still better than harming an innocent person by accident. They already fail in many cases, obviously we accept a failure rate above 0%. You also skipped over the rest of that paragraph.
> users like Chrome and are free to use it and tend to find site protection useful (or at least "not a burden to them")
That's okay, Google can abide by the proposal I set forth avoiding automated mistaken harms to people. If they want to build this system that can do great harms to people, they need to first and foremost build in safety nets to address those harms they cause, and only then focus on reducing false negatives.
I don't think we can easily discard the first in favor of the second. Not nearly as categorically as is done here. Those "false negatives" mean users lose things (bank accounts, privacy, access to their computer) through no fault of their own. We should pause and consider that before weeping and rending our garments that yet another hosting provider solution had a bad day.
You've stopped considering monopoly and correctly considered that the real issue is safe browsing, as a feature, is useful to users and disruptive to new business models. But that's independent of Google; that's the nature of sharing a network between actors that want to provide useful services to people and actors that want to cause harm. If I build a browser today, from scratch, that included safe browsing we'd be in the same place and there'd be no Google in the story.
To be fair, I evaluated that trade off before replying. It's also not just "new sites", but literally any site or person which could be victimized by "safe browsing".
> Those "false negatives" mean users lose things (bank accounts, privacy, access to their computer) through no fault of their own.
That was already happening, and will continue to happen, no matter what. The only thing that the false negative caused is, a stranger didn't swoop in to save a 2nd stranger from a 3rd stranger. That's ok: superheros are bad government. The government should be the one protecting citizens.
Well, no... That's the thing about false negatives vs. true negatives. The more effective the safe browsing protection is, the fewer false negatives. I think we can agree to disagree on where one should tune the knob between minimizing false negatives and minimizing false positives, especially since
a) you have to be doing something pretty unusual to trigger a false positive (such as "setting up an elaborate mechanism to let user-generated content be hosted off of a subdomain you own")
b) there is a workaround once a publisher is aware of the issue.
> The government should be the one protecting citizens.
This seems to be a claim "Safe browsing should be a government institution." I don't immediately disagree, but we must ask ourselves "Which government do we trust with that responsibility?" In America, that's a near-vertical cliff to scale (and it was even before the current government proved a willingness to weaponize its enforcement capacity against speech that should by rights be protected).
If I don't like Chrome safe browsing protection, I can turn it off or change browsers. What do I do if I don't like my government's safe browsing protection? Is it as opt-out as a corporate-provided one is?
That reiterates what I said: the harms happened before, and will continue happening, no matter what. No action will reduce them to 0.
> a) you have to be doing something pretty unusual to trigger a false positive
I don't think that's true here. Many people have been harmed due to trivial, common actions. Other victims, their charges are secret, and they are not afforded due process, an impartial judge, or even the right to face their accusers. Very tyrannic and kafka-esque. Without transparency into the precise rules and process, we categorically cannot make the above claim, and evidence seems to belie it.
> This seems to be a claim "Safe browsing should be a government institution."... What do I do if I don't like my government's safe browsing protection? Is it as opt-out as a corporate-provided one is?
Good news! It isn't. Who says the government needs to provide safe browsing protection? There are other levers governments can take, like investigating and prosecuting criminals, and making victims whole. "Safe browsing" exists because the government has so far failed at that. Law enforcement is more focused on rounding up & perpetrating violence upon people with different skin color than them, I guess.
All that said, I feel like I articulated a pretty good alternative if google really wants to keep safe browsing going: just provide due process to their victims, which includes: a presumption of innocence (one even weaker than in public policy); the right to face their accusers; the right to a speedy, public trial; the right to defend themselves; and the right to an impartial judge/jury.
I gave the grace of assuming you weren't making the absurd argument "You can't ever protect against all ills, therefore you shouldn't try." I won't continue in making that error if that was an error.
> Many people have been harmed due to trivial, common actions
[citation needed]. 100% of safe browsing flaggings I am familiar with are "We let users put content on our site without vetting it," or "We hosted binaries without vetting them and one was malware," or "We got owned and didn't know it." I'm sure there are false positives that are truly false, but I'm aware of zero. Google isn't generally in the business of being preemptive about this sort of thing; they tend to add a site to safe browsing warning only after their crawlers have detected an actual threat behavior. Even in the case of immich.cloud, I don't see any evidence that Immich audited 100% of the *.immich.cloud sites against malware or against users using it intentionally to put up a "This is definitely the Bank of America login page" site with Immich de-facto signing off on the legitimacy of that site.
> they are not afforded due process, an impartial judge, or even the right to face their accusers
I would be in favor of improvements to the restoration process, but there are very good reasons to make addition to the safe browsing list fast; sites on the safe browsing list demonstrated an actual harm vector. Being added to the list isn't being found guilty; it's being arrested by a cop on the street on suspicion of guilt. I will concur that Google is under-incentivized to aggressively crawl red-paged sites to see if they are recovered.
> Very tyrannic and kafka-esque
The key difference is that users may stop using Chrome if it bothers them. Since they don't, I think we can make the educated guess that the benefit outweighs the harm for Chrome users.
> Without transparency into the precise rules and process
Non-starter. The process is a cat-and-mouse against hostile actors. They can't inform the public of all the rules without informing the hostile actors at the same time. This is similar to the reason they don't publish an exhaustive list of what gets an ad banned.
> Who says the government needs to provide safe browsing protection?
Perhaps I misunderstood you. Two posts up: "a stranger didn't swoop in to save a 2nd stranger from a 3rd stranger. That's ok: superheros are bad government. The government should be the one protecting citizens." I thought this meant that it was not okay for private companies to be providing safe browsing protection, but it would be okay for a government to do so? Perhaps the superhero metaphor is lost on me.
> There are other levers governments can take, like investigating and prosecuting criminals, and making victims whole. "Safe browsing" exists because the government has so far failed at that.
Here we are in agreement. But I think rounding up and prosecuting criminals and making victims whole in this context would require sweeping and trans-national law changes to e.g. give a grandmother in Idaho her life savings back after operatives in Russia steal it, with at least two governments conducting inter-country data forensics to resolve the question of who the culprit was that may cost an order of magnitude of resources atop said life savings. I'm not holding my breath (especially since one of those countries is currently under sanctions from the other country).
> just provide due process to their victims, which includes: a presumption of innocence
It's a non-starter. The "victims" here are websites with dodgy reputations and dodgier methods of even contacting them to let them know they look sketchy, much less expectation they will respond to that information. Google red-paging the site is the method of contacting them; the only one that works reliably. The rights you claim necessary are necessary to protect against a government, which one cannot choose to not be a citizen of. Not the safe-access policies of a browser that people can choose not to use.
Your fundamental concern is that the current status quo is tilted towards user safety and against new site admins vs. incumbent sites. Yes. This is the correct place for a browser to put the risk-reward weights. Email has already followed the same pattern for very similar reasons.
p.s: FWIW, Immich tried to switch from immich.cloud to immich.build and immediately tripped over another issue: their SSL certificate was signed for immich.cloud, and it can't validate a site with the TLD immich.build. Independent of all other issues here about safe browsing in general, Immich seems to be demonstrating a spooky lack of understanding about architecting web services from the point of view of multiple features paid for in blood to keep end-users safe, and I don't feel a lot of trust of the project (or its parent, Futo) in general at this time.
Technical alternatives already exist, see for example GNUnet.
Since Mastodon is, fundamentally, a protocol and reference implementation, people can come up with their own enshittified nodes or clients... And then the rest of the ecosystem can respond by just ignoring that work.
Yes, technically Truth Social is a Mastodon node. My Mastodon node doesn't have to care.
Also, plenty of small and medium businesses are doing fine on the internet. You only hear about ones with problems like this. And if these problems become more frequent and public, Google will put more effort into fixing them.
I think the most practical thing we can do is support people and companies who fall through the cracks, by giving them information to understand their situation and recover, and by promoting them.
Why would they do that? Do they lose money from these people? Why would they care? they're a monopoly they don't need to care
For some group of smart people, there will be a group of smarter people who want to dominate the The people they designate "the stupids".
The internet was a technological solution to a social problem. It introduced other social problems, although arguably these to your point are old social problems in a new arena.
But there may be yet another technological solution to the old social problems of monopolism, despotic centralized control, and fraud.
.... I did say "may".
We _still_ have a "smart person only internet" really, it's just now used mostly for drug and weapon sales ( Tor )
I own what I think are the key protocols for the future of browsers and the web, and nobody knows it yet. I'm not committed to forking the web by any means, but I do think I have a once-in-a-generation opportunity to remake the system if I were determined to and knew how to remake it into something better.
If you want to talk more, reach out!
I do think I invite people to disrespect me a little though. It ensures that I have to work harder and succeed on the merit of my work.
These apply to a lot of other decentralized systems too.
It’s actually pretty quick and easy. They cannot defend themselves with lawyers, so a director usually has to show up.
Slam dunk. took all of 6-8 hours of my time end to end. The claim was a single page document. Got the max award allowable. Would have got more had it been California.
55.090 Appearance by parties and attorneys; witnesses. (1) Except as may otherwise be provided by ORS 55.040, no attorney at law nor any person other than the plaintiff and defendant shall become involved in or in any manner interfere with the prosecution or defense of the litigation in the department without the consent of the justice of the justice court, nor shall it be necessary to summon witnesses.
Another idea that's worth investigating are coordinated payment strikes on leveraged companies that offer monthly services like telco companies. A bunch of their customers going "Oops, guess I can't afford to pay this month, gonna have to eat that 2% late fee next month, or maybe the month after that, or maybe the month after that" on a service that won't be disconnected in the first month could absolutely crush a company that requires that monthly income to pay their debt.
[0] https://jacobin.com/2022/05/mass-arbitration-mandatory-agree...
But, good news, it seems like they are walking back on that. They recently ruled that lower courts must "pause" a suit and the suit can resume if an agreement is not made through arbitration.
https://www.bressler.com/news-supreme-court-clarifies-mandat...
I’d run a PnL, get average daily income from visitors, then claim that loss as damages. In court I’d bring a simple spreadsheet showing the hole in income as evidence of damages.
If there were contractors to help get the site back up I’d claim their payments as damages and include their invoices as evidence.
I find it hard to believe that the double redirect itself tripped it: multiple redirects in a row is completely normal—discouraged in general because it hurts performance, but you encounter them all the time. For example, http://foo.example → https://foo.example → https://www.foo.example (http → https, then add or remove www subdomain) is the recommended pattern. And site root to app path to login page is also pretty common. This then leads me to the conclusion that they’re not disclosing what actually tripped it. Maybe multiple redirects contributed to it, a bad learned behaviour in an inscrutable machine learning model perhaps, but it alone is utterly innocuous. There’s something else to it.
There is no responses from Google about this. I had my instance flagged 3 times on 2 different domains including all subdomains, displaying a nice red banner on a representative business website. Cool stuff!
https://govuk-components.netlify.app/
I use Google Workspace for my company email, so that's the only way for me to get in contact with a human, but they refuse to go off script and won't help me contact the actual department responsible in any way.
It's now on a proper domain, https://govuk-components.x-govuk.org/ - but other than moving, there's still not much anyone can do if they're incorrectly targeted.
Some steps to prevent this happening to you:
1. Host only code you own & control on your own domain. Unless...
2. If you have a use-case for allowing arbitrary users to publish & host arbitrary code on a domain you own (or subdomains of), then ensure that domain is a separate dedicated one to the ones you use for your own owned code, that can't be confused with your own owned hosted content.
3. If you're allowing arbitrary members of the public to publish arbitrary code for preview/testing purposes on a domain you own - have the same separation in place for that domain as mentioned above.
4. If you have either of the above two use-cases, publish that separated domain on the Mozilla Public Suffix list https://publicsuffix.org/
The main issue is protecting innocent users from themselves - that's a hard one to generalise solutions to & really depends on your publishing workflows.
Beyond that, the last item (Public Suffix list) comes with some decent additional mitigations as an upside - the main one being that Firefox & Chrome both enable more restrictive cookie settings while browsing any domains listed in the public suffix list.
---
All that said - the question asked in the comment at the top of the thread wasn't about protecting users from security risk, but protecting the domain from being flagged by Google. The above steps should at least do that pretty reliably, barring an actual legitimate hack occurring.
Befriend a lawyer that will agree to send a letter to Google on your behalf in case it happens.
From their perspective, a few false positives over the total number of actual malicious websites blocked is fractional.
Instead, you should be able to install a preferred contentfilter into your browser.
YAML itself is cursed: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...
Perhaps a complaint to the ETC for abusing the monopoly and lack of due process to harm legitimate business? Or DG COMP (in the EU).
Gather evidence of harm and seek alliances with other open-source projects could build a momentum.
Google shouldn’t be a single chokepoint for web censorship.
If you have internal auth testing domains at the same place as user generated content, what's to stop somebody thinking a user-generated page isn't a legit page when it asked you to login or something?
To me this seems like a reasonable flag.
Google Postmaster Console [2] is another one everybody should set up on every domain, even if you don't use gmail. And Google Ads, even if you don't run ads.
I also recommend that people set up Bing search console [3] and some service to monitor DMARC reports.
It's unfortunate that so much of the internet has coalesced around a few private companies, but it's undeniably important to "keep them happy" to make sure your domain's reputation isn't randomly ruined.
[1] https://krebsonsecurity.com/2020/08/why-where-you-should-you...
If any company controls some (high) percentage of a particular market, say web browsers, search, or e-commerce, or social media, the public's equal access should start to look more like a right and less like an at-will contract.
30 years ago, if a shop had a falling out with the landlord, it could move to the next building over and resume business. Now if you annoy eBay, Amazon or Walmart, you're locked out nationwide. If you're an Uber, Lyft, or Doordash (etc) gig worker and their bots decide they don't like you anymore, then sayonara sucker! Your account has been disabled, have a nice day and don't reapply.
Our regulatory structure and economies of scale encourage consolidation and scale and grant access to this market to these businesses, but we aren't protecting the now powerless individuals and small businesses who are randomly and needlessly tossed out with nobody to answer their pleas of desperation, no explanation of rules broken, and no opportunity to appeal with transparency.
It's a sorry state of affairs at the moment.
I suspect the EU will be the first region to push the big tech companies on this.
As firearm enthusiasts like to say, "Enforce the laws we already have".
> No person engaged in commerce or in any activity affecting commerce shall acquire, directly or indirectly, the whole or any part of the stock or other share capital and no person subject to the jurisdiction of the Federal Trade Commission shall acquire the whole or any part of the assets of another person engaged also in commerce or in any activity affecting commerce, where in any line of commerce or in any activity affecting commerce in any section of the country, the effect of such acquisition may be substantially to lessen competition, or to tend to create a monopoly.
Taken at face value, that would forbid companies from buying any large competitors unless the competitor is already failing. Somehow that got watered down into almost nothing.
Instead of taking a consumer-centric / competition perspective, they should be defined in terms of market share (with markets broadly defined from a consumer perspective).
>10% = some minimal interoperability and reporting requirements
>25% = serious interoperability requirements
>35% = severe and audited interoperability requirements, with a method for gaps to be proposed by competitors, with the end goal of making increasing market share past this point difficult
Close the "but it's free to consumers (because we monetize them in other ways)" loophole that every 90s+ internet business used: instead focus on ensuring competition as measured by market share.
It's regulatory capture
now a small competitor wants to do something like get into the wifi game and they're look at huge fixed fees to get started.
If you want to make electronics with any complexity, you'll suddenly discover that you need to pay patent fees. And those come as a fixed share of your revenue. Add enough complexity and you can easily be required to pay more than 100% of your revenue in fees.
Partly a consequence of the biggest tech firms getting bigger.
And partly because of newfound technical ability to achieve mass lock-in (e.g. vendor-controlled encryption, TPMs, vertical integration in platforms, first-party app stores, etc).
The 'but regulatory capture' counter argument rings hollow when the government has given the market a lighter monopoly regulatory touch... and we've ended up with a more concentrated, less competitive market than when it was more heavily regulated.
... It's a lot easier to have a spine about risking getting banned from a service if getting banned from that service wouldn't destroy your life.
https://www.youtube.com/watch?v=mBcY3W5WgNU
But seriously; the internet is now overrun with AI Slop, Spam, and automated traffic. To try to do something about it requires curation, somebody needs to decide what is junk, which is completely antithetical to open protocols. This problem is structurally unsolvable, there is no solution, there's either a useless open internet or a useful closed one. The internet is voting with Cloudflare, Discord, Facebook, to be useful, not open. The alternative is trying to figure out how to run a decentralized dictatorship that only allows good things to happen; a delusion.
The only other solution is accountability, a presence tied to your physical identity; so that an attacker cannot just create 100,000 identities from 25,000 IP addresses and smash your small forum with them. That's an even less popular idea, even though it would make open systems actually possible. Building your own search engine or video platform would be super easy, barely an inconvenience. No need for Cloudflare if the police know who every visitor is. No need for a spam filter, if the government can enforce laws perfectly.
Take a look at email, the mother of all open protocols (older than HTTP). What happened? Radical recentralization to companies that had effective spam management, and now we on HN complain we can't break through, someone needs to do something about that centralization, so that we can go back to square one where people get spammed to death again, which will inevitably repeat the discretion required -> who has the best discretion -> flee there cycle. Go figure.
Gmail processes 376 billion emails per day. At that volume, even 0.1% spam getting through is 376 million messages. However, we're not talking about 0.1%, but 45.6% of email being spam globally. For Gmail, that's 171 billion spam messages daily. Congrats that your private server works at your scale. It's completely irrelevant, and only works because bad actors don't care about it.
Imagine though, if we even accepted spam culturally and handled it individually, as per your solution. That would mean spam can get through with brute force, which it can't right now, meaning that 45.6% would probably explode closer to 90%, 95%, or more overnight. It's only manageable at 45.6% for you because Gmail's spam filters are working overtime harming the economics.
The contra-example, of course, is email. SpamAssassin figured this out 24 years (!) ago. There is zero reason you couldn't apply similar heuristics to detect AI-slop or whatever particular kind of content you don't want to accept.
> Radical recentralization to companies that had effective spam management
Only for the lazy.
B. SpamAssassin is benefiting from centralized players, like Gmail, harming spam's economics. You're a free rider from the onslaught that would occur if spamming actually worked. Spam is at 45.6% of email globally with aggressive spam filters, but could easily double, triple, quadruple in volume if filters started failing even moderately. Weaker filters, and we'll start seeing the email DDoS for the first time.
C. Heuristics on AI Content? What are you going to do, run an "AI Detector" model on a GPU for every incoming email? 376 billion of them every day to Gmail alone? This only makes the email DDoS even more likely.
D. Lazy = 99%+ of global computer users - and that's changing as soon as everyone becomes their own paramedic. If you can't convince most people to learn how to save other people's lives, and probably didn't bother yourself, despite it being disproportionately more important, you're never teaching them technical literacy.
No, you don't need an AI model to detect AI content (lmao). Heuristics already exist, and you see people mention them online all the time -- excessive use of lists, em dashes, common phrases, etc. Yes, a basic text heuristic scorer from the 1980s can pick these up without much difficulty. The magic of auto-learning heuristics (which have also existed since the 1980s, and performed fine at scale with less processing power than your smartwatch) is you can train them on whatever content you don't want to receive: marketing, political content, etc. You can absolutely apply this to whatever content suits your fancy, and it doesn't really take any more effort than moving messages you want filtered out to a Junk folder or similar.
The issues with todays internet stem specifically from the centralisation of power in the hands of Google, Apple and the social networks.
Bad search results? Blame Google's monopoly incentivising them intentionally making their results worse.
Difficulty promoting or finding events? Blame Facebooks real revenue model - preventing one to many communications by default and charging for exceptions.
AI overrun with slop? Blame OpenAI and Facebook, both of whom are actively promoting and profiting from the creation of slop.
Automated traffic slowing down sites? It's often the AI companies indexing and reindexing hundreds of times.
Spam? Not a huge issue for anyone that I'm aware of.
The closed internet platforms are the problem. Forcing them to relinquish control over handsets, data and our interpersonal connections is the solution. It will be legislative, or it will be torches to the data centres, likely both. But it is coming.
> The issues with todays internet stem specifically from the centralisation of power in the hands of Google and the social networks.
> Bad search results? Blame Google's monopoly incentivising them intentionally making their results worse.
> Difficulty promoting or finding events? Blame Facebooks real revenue model - preventing one to many communications by default and charging for exceptions.
You're misdiagnosing what happened here. These aren't diseases. These are symptoms that the more open internet, that we had in the early 2000s, completely failed at scale. The disease was the predictable failure of an open system to self-moderate, the symptoms the centralization that followed. You're mistaking effect for cause.
People started using Google, because it was the only tool good enough at digging through manure. Facebook started charging for mass communication, because otherwise, everyone has an excuse why they need to use it. Cloudflare became popular, because the internet didn't care when 40% of traffic was bots, half of them malicious, before AI was even on the scene. And so on.
The open system failed, and was becoming unusable. Big Tech arrived offering proprietary solutions as CPR. They didn't cause the death.
First off, we can simply let the user, or client software, choose. Why should we let centralized servers do that by default?
At scale, DNS is somewhat centralized but authorities are disconnected from internet providers and web browsers. They're the best actors to regulate this.
For mail, couldn't we come up with a mail-DNS, that authenticates senders? There could be different limits based on whether you are an individual or a company, and whether you're sending 10'000 emails or just 100.
Regardless of whether these are good solutions -- why jump to extreme ones? "TINA" is not a helpful argument, it's a slogan.
So RFC 7672? https://datatracker.ietf.org/doc/html/rfc7672
Also, I think it solves a different problem: it prevents spoofing/MITM but what about legitimate certificates? We would still need CAs that actually curate their customers and hold them accountable. And we would need email servers/clients to differentiate between strict CAs and ones that are used solely for encryption purposes.
I don't know that DNS should be applied to emails as is anyway but I find it could force spammers to operate with publicly available information which would make holding them accountable easier.
It's not hard to set up DNSSEC as long as your DNS server software supports it and most people don't run their own authorative DNS servers anyway.
Supposedly, DMA should enforce this already.
https://www.socialmediatoday.com/news/meta-announces-next-st...
Haven't heard much about it lately though.
We do monthly updates on the status of the project that we call State of the Bird and they can be found here https://discourse.imfreedom.org/tag/state-of-the-bird.
Maybe they should be subject to same limitations like First Amendment etc.
It's a bit frustrating when a company becomes a major player in an industry and can have a life and death sentence on other businesses.
There are alternative payment method but people are use to pay a certain way in that industry/area, similarly there are other browsers but people are used to Chrome.
fuck paypal
High quality or even medium quality software and UX is getting harder and harder to find.
At the time I didn't have a better way to test that my form worked.
Fwiw Venmo is run by the same thugs who run PayPal. So go figure.
Randomly, years later, they turned it back on. Thanks, I guess?
Google flagged my domain as dangerous once. I do host Jellyfin, Immish, and NextCloud. I run an IP whitelist on the router. All packets from IPs that are not whitelisted are dropped. There are no links to my domain on the internet. At any time, there are 2-3 IPs belonging to me and my family that can load the website. I never whitelisted Google IPs.
How on earth did Google manage to determine that my domain is dangerous?
At this point I would rather use an analog camera with photo albums than Google Photos.
Fortunately, I expose it to the internet on its own domain despite running through the same reverse proxy as other projects. It would have sucked if this had happened to a domain used for anything else, since the appeal process is completely opaque.
Just because it's your website, and you're not a bad agent doesn't prove that no part of the site is under the control of a bad agent, and that your site isn't accidentally hosting something malicious somewhere, or have some UI that is exploitable for cross-site scripting or whatever.
Navigating to https://main.preview.internal.immich.cloud, I'm right away informed by the browser that the connection is not secure due to an issue with the certificate. The problem is that it has the following CN (common name): main.preview.internal.immich.build. The list of alternative names also contains that same domain name. It does not match the site: the certificate's TLD .build is different from the site's .cloud!
I don't see the same problem on external sites like tiles.immich.cloud. That has a CN=immich.cloud with tiles.immich.cloud as an alternative.
We still don't know what caused it because it happened to the Cloudflare R2 subdomain, and none of the Search Console verification methods work with R2. It also means it's impossible to request verification.
This is the way of things.
Of course Google will claim it's just a mistake, but large tech companies - including e.g. Microsoft - have indulged in such behavior before. A lawsuit will allow for discovery which can help settle the matter, and may also encourage Google to behave like a good citizen.
Chrome is to Web what Teams is to Chat. Bad job guys.
donmcronald•3mo ago
https://old.reddit.com/r/immich/comments/1oby8fq/immich_is_a...
I had my personal domain I use for self-hosting flagged. I've had the domain for 25 years and it's never had a hint of spam, phishing, or even unintentional issues like compromised sites / services.
It's impossible to know what Google's black box is doing, but, in my case, I suspect my flagging was the result of failing to use a large email provider. I use MXRoute for locally hosted services and network devices because they do a better job of giving me simple, hard limits for sending accounts. That way if anything I have ever gets compromised, the damage in terms of spam will be limited to (ex) 10 messages every 24h.
I invited my sister to a shared Immich album a couple days ago, so I'm guessing that GMail scanned the email notifying her, used the contents + some kind of not-google-or-microsoft sender penalty, and flagged the message as potential spam or phishing. From there, I'd assume the linked domain gets pushed into another system that eventually decides they should blacklist the whole domain.
The thing that really pisses me off is that I just received an email in reply to my request for review and the whole thing is a gas-lighting extravaganza. Google systems indicate your domain no longer contains harmful links or downloads. Keep yourself safe in the future by blah blah blah blah.
Umm. No! It's actually Google's crappy, non-deterministic, careless detection that's flagging my legitimate resources as malicious. Then I have to spend my time running it down and double checking everything before submitting a request to have the false positive mistake on Google's end fixed.
Convince me that Google won't abuse this to make self hosting unbearable.
foobarian•3mo ago
akerl_•3mo ago
This seems like the flagging was a result of the same login page detection that the Immich blog post is referencing? What makes you think it's tied to self-hosted email?
donmcronald•3mo ago
In my case, the Google Search Console explicitly listed the exact URL for a newly created shared album as the cause.
https://photos.example.com/albums/xxxxxxxx-xxxx-xxxx-xxxx-xx...
I wish I would have taken a screenshot. That URL is not going to be guessed randomly and the URL was only transmitted once to one person via e-mail. The sending was done via MXRoute and the recipient was using GMail (legacy Workspace).
The only possible way for Google to have gotten that URL to start the process would have been by scanning the recipient's e-mail. What I was trying to say is that the only way it makes sense to me is if Google via GMail categorized that email as phishing and that kicked off the process to add my domain to the block list.
So, if email categorization / filtering is being used as a heuristic for discovering URLs for the block list, it's possible Google's discriminating against domains that use smaller email hosts that Google doesn't trust as much as themselves, Microsoft, etc..
All around it sucks and Google shouldn't be allowed to use non-deterministic guesswork to put domains on a block list that has a significant negative impact. If they want to operate a clown show like that, they should at least be liable for the outcomes IMO.
david_van_loon•3mo ago
It's scary how much control Google has over which content people can access on the web - or even on their local network!
Larrikin•3mo ago
donmcronald•3mo ago