There is this weird divide between the certified class of non-technical consultants and actual overworked and pushed to corner cut techs.
From what I have seen a lot of time the playbooks to fix these issues are just rawdogging files using rsync manually. Ideally you deploy your infrastructure in cells where rollouts proceed cell by cell so you can catch issues sooner and also implement failover to bootstrap broken cells (in my DNS example, client could talk to DNS servers in the closest non-broken cell using BGP based routing). It is hard to test, and there are some global services (like that big Google outage a few months ago was due to the global auth service being down).
> "The outage also hit servers that host procedures meant to overcome such an outage... Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored."
https://www.reuters.com/technology/space/power-failed-spacex...
So confidentiality was maintained but integrity and availability were not.
Yes, it's fucking expensive, that's one of the reason you pay more for a VM (or colocation) than at Hetzner or OVH. But I'm also pretty confident that single fire wouldn't destroy all hard drives in that IT space.
Yes, the servers still have some small batteries on their mainboards etc, but it's not too bad.
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
Does G-Drive mean Google Drive, or "the drive you see as G:"?
If this is Google Drive, what they had locally were just pointers (for native Google Drive docs), or synchronized documents.
If this means the letter a network disk storage system was mapped to, this is a weird way of presenting the problem (I am typing on the black keyboard and the wooden table, so that you know)
But yeah it's a big problem in Korea right now, lots of important information just vanished, many are talking about it.
Sometimes things can seem to run smoothly for years when neglected... until they suddenly no longer run smoothly!
Electronically, everyone just receives a link to read the document.
https://www.nytimes.com/2025/09/13/world/asia/nepal-unrest-a... ("Many of the nation’s public records were destroyed in the arson strikes, complicating efforts to provide basic health care")
TL;DR: Estonia operates a Tier 4 (highest security) data center in Luxembourg with diplomatic immunity. Can actively run critical government services in real-time, not just backups.
> The actual number of users is about 17% of all central government officials
Far from all, and they're not sure what's recoverable yet ("“It’s difficult to determine exactly what data has been lost.”")
Which is not to say that it's not big news ("the damage to small business owners who have entered amounts to 12.6 billion Korean won.” The ‘National Happiness Card,’ used for paying childcare fees, etc., is still ‘non-functional.’"), but to put it a bit in perspective and not just "all was lost" as the original submission basically stated
Quotes from https://www.chosun.com/english/national-en/2025/10/02/FPWGFS... as linked by u/layer8 elsewhere in this thread
Without an out-of-country backup, a reversion to previous statuses means the country is lost (Estonia has been occupied a lot). With it, much of the government can continue to function, as an expat government until freedom and independence is restored.
hmmmm
I wish the same concept was in Canada as well. You absolutely have to resubmit all your information every time you do a request. On top of that, federal government agencies still mail each other the information, so what usually can be done in 1 day takes a whole month to process, assuming the mail post isn't on strike (spoiler: they are now).
I think Canada is one of the worst countries in efficiency and useless bureaucracy among 1st world countries.
This is the state of banking in Canada. God forbid they just put a text box on the banking web app where I can put in my beneficiary.
Not to mention our entire health care system still runs on fax!
It blows my mind that we have some of the smartest and well educated people in the world with some of the highest gdp per capita in the world and we cannot figure out how to get rid of paper documents. You should be issued a federal digital ID at birth which is attested through a chain of trust back to the federal government. Everything related to the government should be tied back to that ID.
This attempt at putting it in perspective makes me wonder what would put it in perspective. "100M sets of harry potter novels" would be one step in the right direction, but nobody can imagine 100M of anything either. Something like "a million movies" wouldn't work because they are very different from text media in terms of how much information is in one, even if the bulk of the data is likely media. It's an interesting problem even if this article's attempt is so bad it's almost funny
Good article otherwise though, indeed a lot more detail than the OP. It should probably replace the submission. Edit: dang was 1 minute faster than me :)
This is why I don't really want to run my own cloud :)
Actually testing the backups is boring.
That said, ones the flames are out, they might actually be able to recover some of it.
Idk if this sounds like I'm against backups, I'm not, I'm just surprised by the question
[0]: https://www.cnbc.com/2025/02/13/company-ripped-by-elon-musk-...
buttons are jpegs/gifs, everything is on Java EE and on vulnerable old webservers etc... A lot of government stuff supports only Internet Explorer even though it's long dead
Don't even get me started on ActiveX.
You're thinking of Taiwan, not South Korea.
https://m.blog.naver.com/gard7251/221339784832 (a random blog with gifs)
They also require routine testing distaster recovery plans.
I participated in so many different programs over the years with those tests.
Tests that would roll over to facilities across the country
They very well might have only been saving to this storage system. It was probably mapped as a drive or shared folder on the PC.
Like, I use Google Drive for Desktop but it only downloads the files I access. If I don't touch a file for a few days it's removed from my local cache.
Who has the incentive to do this, though? China/North Korea? Or someone in South Korea trying to cover up how bad they messed up? Does adding this additional mess on top mean they looked like they messed up less? (And for that to be true, how horrifically bad does the hack have to be?)
Not saying I believe this (or even know enough to have an opinion), but it’s always important to not anthropomorphize a large organization. The government isn’t one person (even in totalitarian societies) but an organization that contains large numbers of people who may all have their own motivations.
Alternate hypothesis: cloud storage provided doing the hard sell. Hahaha :)
LG is SK firm and manufacturer of hacked hardware and also the batteries that caught fire. Not sure it’s a solid theory just something I took note of while thinking the same
Yeah, that's way less suspicious, thanks for clearing that up.
> 27th of September 2025, The fire is believed to have been caused while replacing Lithium-ion batteries. The batteries were manufactured by LG, the parent company of LG Uplus (the one that got hacked by the APT).
Could the battery firmware have been sabotaged by the hacker to start the fire?
But
replacing a UPS is usually done to right time pressures. the problem is, you can rarely de-energise UPS batteries before replacing them, you just need to be really careful when you do it.
Depending on the UPS, Bus bars can be a mother fucker to get on, and of they touch energised they tend to weld together.
With lead acid, its pretty bad (think molten metal and lots of acidic, toxic and explosive gas, with lithium, its just fire. lots of fire that is really really hard to put out.
Obviously for rack based UPSs you'd "just" take out the UPS, or battery drawer, and replace somewhere more safe, or better yet, swap out the entire thing.
For more centralised UPSs that gets more difficult. The shitty old large UPSs were a bunch of cells bolted to a bus bar, and then onto the switchgear/concentraitor.
for Lithium, I would hope its proper electrical connectors, but you can never really tell.
A Kakao datacenter fire took the de-facto national chat app offline not too many years ago. Imagine operating a service that was nearly ubiquitous in the state of California and not being able to survive one datacenter outage.
After reading the Phrack article, I don't know what to suspect, the typical IT disaster preparedness or the operators turning off the fire suppression main and ordering anyone in the room to evacuate to give a little UPS fire enough time to start going cabinet to cabinet.
Recently in the UK a major communication company had issues with batteries
Bravo, old boy.
Anyway, shoe production has never been better.
How do we know it's NK? Because there are comments in north-korean language, duh! Why are you asking, are you russian bot or smt??
The bad news is: they're in North Korea.
"But that's a blessed event? How could that be terrible?"
"He appeared in Salt Lake City."
It's just in NK or china.
Yikes.
This was also often tied to a big counter culture movement. Which one interesting thing is that many of those people now define the culture. I guess not too unlike how many hippies changed when they grew up
Etymology quibble: There is no 'c' in phreaking. Phrack is just a portmanteau of "phreak" and "hack". :)
> Phrack is just a portmanteau of "phreak" and "hack". :)
You delete your data, trash your gear, and hop on a bus, to start over in some other city, in a different line of work.
Compromised batteries or battery controllers?
This story is really unbelievable.
I'm guessing lithium-ion batteries were not a factor years ago when those decisions were made.
As the other commenter said, Halon hasn't been a thing for a fair while, but inert gas fire suppression systems in general are still popular.
I would expect it wouldn't be sufficient for a lithium ion battery fire - you'd temporarily displace the oxygen, sure, but the conditions for fire would still exist - as soon as enough nitrogen (or whatever suppressant gas is in use) dissipates, it'd start back up again.
Also as I understand thermal runaway is self-sustaining, since the lithium ion batteries have a limited capacity to provide their own oxygen (something to do with the cathode breaking down?), so it might continue burning even while the area is mostly flooded with inert gas.
I believe it would be similar to an EV car fire, that is, you'd have to flood the area with water and wait for it to cool down enough that thermal runaway stops. Maybe they can do better these days with encapsulating agents but I'd still expect the rack housing the UPS to be a write-off.
(Edit: sorry, in hindsight it's obvious the comment I'm replying to was referring to inert gas systems, and not halogenated systems)
Halon and friends don't work through an oxygen displacement mechanism, their fire suppression effects are primarily due to how the halogen moieties interfere with the decomposition of other substances in the flame. IIRC, A key mechanism is the formation of hydrogen(!) from hydrogen radicals.
Apparently if the calibration is correct, halon can de deployed in a space to suppress a fire without posing as asphyxiation risk.
A good review is here: https://www.nist.gov/system/files/documents/el/fire_research...
96 servers isn’t that many, probably less than 10 racks and given the state of the backups it would track that they didn’t spring for halon.
The timing as well is very suspicious and I think that there can be a lot of discussion about this
Right now, I am wondering about the name most tbh which might seem silly but "APT down - The North Korean files"
It seems that APT means in this case advanced persistent threat but I am not sure what they mean by Apt Down, like the fact that it got shut down by their journalism or-? I am sorry if this may seem naive and on a serious note this raises so many questions...
https://www.thetakeout.com/1789352/korea-apt-drinking-game-r...
2 HN front page articles in 1!
Thankfully I made the right choice to stay on Bitwarden instead of moving to Proton Pass.
This deserves its own HN submission. I submitted it but it was flagged due to the title.
Thank you for sharing it on HN.
7 things all kids need to hear
1 I love you
2 I'm proud of you
3 I'm sorry
4 I forgive you
5 I'm listening
6 RAID is not backup. Make offsite backups. Verify backup. Find out restore time. Otherwise, you got what we call Schrödinger backup
7 You've got what it takes
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There’s not half the files there used to be,
And there’s a deadline
hanging over me.
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now my data’s gone
and I long for yesterday-ay-ay-ay.
Yesterday,
The need for back-ups seemed so far away.
Thought all my data was here to stay,
Now I believe in yesterday.multiple copies; multiple locations; multiple formats.
ah the so called schrodingers drive. It's there unless you try to copy it
the destruction of the library of alexandria is under dispute.
His was supposedly laid off years ago, but nobody actually stopped his paycheck, so he kept coming in to work assuming he was still employed, getting shuffled into increasingly-abusive working environments by callously indifferent managers who assume he's somebody else's problem.
"The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups."
This is absolutely wild.
And are we forgetting that they can literally have a multi cloud backup setup in their own country as well or incentivize companies to build their datacenters there in partnership with them of sorts with a multi cloud setup as I said earlier?
Tools like Kopia, Borg and Restic handle this and also include deduplication and other advanced features.
Really no excuse for large orgs or even small businesses and somewhat tech literate public.
> ... a countries' government entire data?
But the bulk of the data is "boring": important to individuals, but not state security ("sorry Jiyeong, the computer doesn't know if you are a government employee. Apologies if you have rent to make this month!")
There likely exists data where the risk calculation ends up differently, so that you wouldn't store it in this system. For example, for nuke launch codes, they might rather lose than loose them. Better to risk having to reset and re-arm them than to have them hijacked
> Is encryption, [in?] any form, really reliable protection
There's always residual risk. E.g.: can you guarantee that every set of guards that you have watching national datacenters is immune from being bribed?
Copying data around on your own territory thus also carries risks, but you cannot get around it if you want backups for (parts of) the data
People in this thread are discussing specific cryptographic primitives that they think are trustworthy, which I think goes a bit deeper than makes sense here. Readily evident is that there are ciphers trusted by different governments around the world for their communication and storage, and that you can layer them such that all need to be broken before arriving at the plain, original data. There is also evidence in the Snowden archives that (iirc) e.g. PGP could not be broken by the NSA at the time. Several ciphers held up for the last 25+ years and are not expected to be broken by quantum computers either. All of these sources can be drawn upon to arrive at a solid choice for an encryption scheme
That's just from the top of my head. Exploiting such a trove of data doesn't sound complicated.
Sometimes sensitive data at the government level has a pretty long shelf life; you may want it to remain secret in 30, 50, 70 years.
https://www.nbcnewyork.com/news/local/north-korea-parking-ti... [2017]
A data breach, however, is completely secret - both from you and from others. Another country (not even necessarily the one that is physically hosting your data) may have access to your data, and neither you nor anyone else would necessarily know.
I am currently evaluating between GCP and AWS right now.
Google Cloud is supposed to have safeguards that don't allow account deletion, but none of them worked apparently, and the only option was a restore from a separate cloud provider (shoutout to the hero at UniSuper who chose a multi-cloud solution).
If you are working with really important software, please follow the 3-2-1 EVEN with cloud providers I suppose if you genuinely want ABSOLUTE guarantee I suppose, but it depends on how important the data is I suppose for the prices.
I have thought about using some cheap like backblaze and wasabi and others for the 3-2-1 for backups I suppose I am not sure but I do think that this incident was definitely a bit interesting to read into and I will read more about it, I do remember it from kevin fang's video but this article is seriously good and I will read it later, bookmarked.
not losing data and keeping untrusted parties out of your data is a hard problem, that "cloud" aka "stored somewhere that is accessible by agents of a foreign nation" does not solve.
As a government you should not be putting your stuff in an environment under control of some other nation, period. That is a completely different issue and does not really relate to making backups.
Except for the backup strategy said consumers apply to their data themselves, right?
If I use a service called “it is stored in a datacenter in Virginia” then I will not be surprised when the meteor that hits Virginia destroys my data. For that reason I might also store copies of important things using the “it is stored in a datacenter in Oregon” service or something.
Amazon S3 provides a highly durable storage infrastructure designed for mission-critical and primary data storage. S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.
You can save a little money by giving up that redundancy and having your data i a single AZ:
The S3 One Zone-IA storage class stores data redundantly across multiple devices within a single Availability Zone
For further redundancy you can set up replication to another region, but if I needed that level of redundancy, I'd probably store another copy of data with a different cloud provider so an AWS global failure (or more likely, a billing issue) doesn't leave my data trapped in one vendor).
I believe Google and Azure have similar levels of redundancy levels in their cloud storage.
You backup stuff. To other regions.
> GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters.
Of course, it's perfectly possible to have proper distributed storage without using a cloud provider. It happens to be hard to implement correctly, so apparently, the SK government team in question just decided... not to?
Why? If you encrypt it yourself before transfer, the only possible control some_other_nation will have over you or your data is availability.
You don't need the Cloud for backups, and there's no reason to believe that they would have backuped their data while using the cloud more than what they did with their self-hosting…
Sovereign delivery makes sense for _nations_.
They have 2 Trillion $ economy. I am sure that competency shouldn't be the thing that they should be worrying at that scale but at the same time I know those 2 trillion $ don't really make them more competent but I just want to share that it was very possible for them to teach/learn the competency
Maybe this incident teaches us atleast something. Definitely something to learn here though. I am interested in how the parent comment suggests sharing one time pad or rather a practical way for them to do so I suppose since I am genuinely curious as most others refer to using the cloud like aws etc. and I am not sure how much they can share something like one time pad and at the scale of petabytes and more, I can maybe understand it but I would love if the GP can tell me a practical way of doing so to atleast have more safety I suppose than encryption methods I suppose..
It could be a gov laptop with the encryption keys left at a bar. Or the wrong keys saved on the system and the backups can't actually be decrypted. Or the keys being reused at large scale and leaked/guessed from lower security area. etc.
Relying on encryption requires operation knowledge and discipline. At some point, a base level of competency is required anyway, I'm not just sure encryption would have saved them as much as we'd wish it would.
To your point, I'd assume high profile incidents like this one will put more pressure to do radical changes, and in particular to treat digital data as a more critical asset that you can't hand down to the crookest corrupt entity willy nilly just for the kickback.
South Korea doesn't lack competent people, but hiring them and letting them at the helm sounds like a tough task.
To add, use F710s, H710s and then add ObjectScale storage for your Kubernetes workloads.
This setup repatriates your data and gives you a Cloud like experience. Pair it with like EKS-A and you have a really good on premises Private Cloud that is resilient.
For those there's sure no problem
Like, don't store it in the cloud of an enemy country of course.
But if it's encrypted and you're keeping a live backup in a second country with a second company, ideally with a different geopolitical alignment, I don't see the problem.
https://www.huffpost.com/entry/south-korea-human-rights-inve...
How Trump’s ICE Raid Triggered Nationwide Outrage in South Korea
https://www.newsweek.com/trump-ice-raid-hyundai-outrage-sout...
'The raid "will do lasting damage to America's credibility," John Delury, a senior fellow at the Asia Society think tank, told Bloomberg. "How can a government that treats Koreans this way be relied upon as an 'ironclad' ally in a crisis?"'
I'm sure cryptographers would love to know what makes it possible for you to assume that say AES-256 or AES-512 can be broken in practice for you to include it in your risk assessment.
But you could also go with something like OTP and then it's actually fundamentally unbreakable. If the data truly is that important, surely double the storage cost would also be worth it.
Exfiltrating terabytes of data is difficult, exfiltrating 32 bytes is much less so.
Huh? An enemy country will shut off your access. Friendly countries don't.
> Even if it's encrypted data, all encryption can be broken, and so we must assume it will be broken.
This is a very, very hot take.
you are seeing the local storage decision under the lens of security, that is not the real reason for this type of decision.
While it may have been sold that way, reality is more likely the local DC companies just lobbied for it to be kept local and cut as many corners as they needed. Both the fire and architecture show they did cut deeply.
Now why would a local company voluntary cut down its share of the pie by suggesting to backup store in a foreign country. They are going to suggest keep in country or worse as was done here literally the same facility and save/make even more !
The civil service would also prefer everything local either for nationalistic /economic reasons or if corrupt then for all kick backs each step of the way, first for the contract, next for the building permits, utilities and so on.
But while countries go from unfriendly to attacking you overnight, they don't generally go from friendly to attacking you overnight.
If the US wants its state-puppet corporations to be used for integral infrastructure by foreign governments, it's going to need to provide some better legal assurances than 'trust me bro'.
(Some laws on the books, and a congress and a SCOTUS that has demonstrated a willingness to enforce those laws against a rogue executive would be a good start.)
America is not a stable ally, and has a history of spying on friends.
So unless the whole of your backup is encrypted offline, and you trust the NSA to never break the encryption you chose, its a national security risk.
The lack of a backdoor can be proven by choosing parameters according to straightforward reasons that do not allow the possibility for the chooser to insert a backdoor. The curve25519 parameters have good reasons why they are chosen. By contrast, Dual_EC_DRBG contains two random-looking numbers, which the NSA pinky-swears were completely random, but actually they generated them using a private key that only the NSA knows. Since the NSA got to choose any numbers to fit there, they could do that. When something is, like, "the greatest prime number less than 2^255" you can't just insert the public key of your private key into that slot because the chance the NSA can generate a private key whose public key just happens to match the greatest prime number less than 2^255 is zero. These are called "nothing up my sleeve numbers".
This doesn't prove the algorithm isn't just plain old weak, but nobody's been able to break it, either. Or find any reason why it would be breakable. Elliptic curves being unbreakable rests on the discrete logarithm of a random-looking permutation being impossible to efficiently solve, in a similar way to how RSA being unbreakable relies on nobody being able to efficiently factorize very big numbers. The best known algorithms for solving discrete logarithm require O(sqrt(n)) time, so you get half the bits of security as the length of the numbers involved; a 256-bit curve offers 128 bits of security, which is generally considered sufficient.
(Unlike RSA, you can't just arbitrarily increase the bit length but have to choose a completely new curve for each bit length, unfortunately. ed25519 will always be 255 bits, and if a different length is needed, it'll be similar but called something else. On the other hand, that makes it very easy to standardize.)
Absence of evidence is not evidence of absence. It could well be that someone has been able to break it but that they or that organization did not publish.
I agree on the evidence/absence of conjecture. However, the impact of the secret feels impossible to keep.
Time will, of course, tell; it wouldn't be the first occasion where that has embarrassed me.
America is a shitty ally for many reasons. But spying on allies isn’t one of them. Allies spy on allies to verify they’re still allies. This has been done throughout history and is basic competency in statecraft.
No nation should trust the USA, especially not with their state secrets, if they can help it. Not that other countries are inherently more trustworthy, but the US is a known bad actor.
We also know this is also true for Russia, China and India. Being spied on is part of the cost of relying on external security guarantees.
> Not that other countries are inherently more trustworthy, but the US is a known bad actor
All regional and global powers are known bad actors. That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.
Case in point: https://en.wikipedia.org/wiki/2025_Georgia_Hyundai_plant_imm...
> The raid led to a diplomatic dispute between the United States and South Korea, with over 300 Koreans detained, and increased concerns about foreign companies investing in the United States.
That they are. Americans tend to view themselves as "the good guys" however, which is a wrong observation and thus needs pointing out in particular.
> That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.
If they have any secrets they attempt to keep even from Washington, they are contained in these backups. If that is the case, storing them (even encrypted) with an American company absolutely compromises security, even if there is no known threat vector at this time. The moment you give up control of your data, it will forever be subject to new threats discovered afterward. And that may just be something like observing the data volume after an event occurs that might give something away.
You really have no evidence to back up your assertion, because you’d have to be an insider.
Perhaps is doing a lot of work here. They do, and they are. That is what the Snowden leaks proved.
> You really have no evidence to back up your assertion, because you’d have to be an insider.
I don't, because the possibility alone warrants the additional caution.
> Bernard Squarcini, head of the Direction Centrale du Renseignement Intérieur (DCRI) intelligence service until last year, told French daily Le Figaro he was “astonished” when Prime Minister Jean-Marc Ayrault said he was "deeply shocked" by the claims.
> “I am amazed by such disconcerting naiveté,” he said in the interview. “You’d almost think our politicians don’t bother to read the reports they get from the intelligence services.”
> “The French intelligence services know full well that all countries, whether or not they are allies in the fight against terrorism, spy on each other all the time,” he said.
> “The Americans spy on French commercial and industrial interests, and we do the same to them because it’s in the national interest to protect our companies.”
> “There was nothing of any real surprise in this report,” he added. “No one is fooled.”
I always thought it was a little unusual that the state of France owns over 25% of the defense and cyber security company Thales.
Unusual from an American perspective, maybe. The French state has stakes in many companies, particularly in critical markets that affect national sovereignty and security, such as defence or energy. There is a government agency to manage this: https://en.wikipedia.org/wiki/Agence_des_participations_de_l... .
People who don’t know history think spying on allies is bad.
https://kakaocloud.com/ https://www.nhncloud.com/ https://cloud.kt.com/
To name a few.
Korea is great at a lot of engineering disciplines. Sadly, software is not one of them, though it's slowly changing. There was a similar issue a couple years ago where the government's internal intranet was down a couple days because someone deployed a switch in front of outbound connections without anyone noticing.
It's not a talent problem but a management problem - similar to Japan's issues, which is unsurprising as Korean institutions and organizations are heavily based on Japanese ones from back in the JETRO era.
> South Korean corporate management is stuffed full of ex-military officers
Absolutely not. It was very common in Germany to deny military service and instead do a year of civil service as a replacement. Also, there were several exceptions from the """mandatory""" military service. I have two brothers who had served, so all I did was tick a checkbox and I was done with the topic of military service.
As somebody that has also done work in Korea (with on of their banks), my observation was that almost all decision making was top-down, and people were forced to do a ton of monotonous work based on the whims of upper management, and people below could not talk back. I literally stood and watched a director walk in after racking a bunch of equipment and commented that the disk arrays should be higher up. When I asked why (they were at the bottom for weight and centre of gravity reasons), he looked shocked that I even asked and tersely said that the blinking lights of the disks at eye level show the value of the purchase better.
I can't imagine writing software in that kind of environment. It'd be almost impossible to do clean work, and even if you did it'd get interfered with. On top of that nobody could go home before the boss.
I did enjoy the fact that the younger Koreans we were working with asked me and my colleague how old we were, because my colleague was 10 years older than me and they were flabbergasted that I was not deferring to him in every conversation, even though we were both equals professionally.
This was circa 2010, so maybe things are better, but oh my god I'm glad it was business trips and I was happy to be flying home each time (though my mouth still waters at the marinaded beef at the bbq restaurants I went to...).
The big problem from my point of view is management. Everyone pushes responsibility and work all the way down to the developera so that they do basically everything themselves from negotiating with the customer, writing the requirements (or not) to designing the architecture, writing the code and testing the system.
If they're late,they just stay and work longer and on the weekends and sleep at the desk.
This is the only part that sounds bad? Negotiating with customers may require some help as well but it's better than having many layers in between.
> Korea is great at a lot of engineering disciplines. Sadly, software is not one of them
NHN Cloud is in fact being used more and more in the government [1], as well as playing a big part in the recovery effort of this fire. [2]
No, unlike what you're suggesting, Korea has plenty of independent domestic cloud and the government has been adopting it more and more. It's not on the level of China, Russia or obviously the US, but it's very much there and accelerating quickly. Incomparable to places like the EU which still have almost nothing.
[1] https://www.ajunews.com/view/20221017140755363 - 2022, will have grown a lot now [2] https://www.mt.co.kr/policy/2025/10/01/2025100110371768374
My head hurts
Not that they should just go all in on it, but an encrypted copy on S3 or GCS would seem really useful right about now.
Cost wise probably having even a Korean different data center backup would not have been huge effort, but not doing it exposed them to a huge risk.
...in an ideal world anyway, in practice I've never seen a disaster recovery training. I've had fire drills plenty of times though.
Primary? No. Back-up?
These guys couldn’t provision a back-up for their on-site data. Why do you think it was competently encrypted?
Doesn't have to be an American provider (Though anyone else probably increases Seoul's security cross section. America is already its security guarantor, with tens of thousands of troops stationed in Korea.)
And doesn't have to be permanent. Ship encrypted copies to S3 while you get your hardenede-bunker domestic option constructed. Still beats the mess that's about to come for South Korea's population.
Using a commercial provider is not a guarantee.
Permanently losing data at a given store point isn't relevant to losing data overall. Data store failures are assumed or else there'd be no point in backups. What matters is whether failures in multiple points happen at the same time, which means a major issue is whether "independent" repositories are actually truly independent or whether (and to what extent) they have some degree of correlation. Using one or more completely unique systems done by someone else entirely is a pretty darn good way to bury accidental correlations with your own stuff, including human factors like the same tech people making the same sorts of mistakes or reusing the same components (software, hardware or both). For government that also includes political factors (like any push towards using purely domestic components).
>They simply should have made local and off-line backups
FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.
I can't reveal any details but it was a lot more than just a given storage point. The interesting thing is that there were multiple points along the way where the damage would have been recoverable but their absolute incompetence made matters much worse to the point where there were no options left.
> FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.
If you can't do the job you should get out of the kitchen.
(If IBM was also running the local storage then we're talking about a very different risk profile from "run your own storage, back up to a cloud" and the anecdote is worth noting but not directly relevant.)
A quick search reveals IBM does still sell backup solutions, including ones that backup from multiple cloud locations and can restore to multiple distinct cloud locations while maintaining high availability.
So, if the claims are true, then IBM screwed up badly.
Sorry, not brain not really clicking tonight and used lazy imprecise terminology here, been a long one. But what I meant by "store point" was any single data repository that can be interacted with as a unit, regardless of implementation details, that's part of a holistic data storage strategy. So in this case the entirety of IBM would be a "storage point", and then your own self-hosted system would be another, and if you also had data replicated to AWS etc those would be others. IBM (or any other cloud storage provider operating in this role) effectively might as well simply be another hard drive. A very big, complex and pricey magic hard drive that can scale its own storage and performance on demand granted, but still a "hard drive".
And hard drives fail, and that's ok. Regardless of the internal details of how the IBM-HDD ended up failing, the only way it'd affect the overall data is if that failure happened simultaneously with enough other failures at local-HDD and AWD-HDD and rsync.net-HDD and GC-HDD etc etc that it exceeded available parity to rebuild. If these are all mirrors, then only simultaneous failure of every single last one of them would do it. It's fine for every single last one of them to fail... just separately, with enough of a time delta between each one that the data can be rebuilt on another.
>If you can't do the job you should get out of the kitchen.
Isn't that precisely what bringing in external entities as part of your infrastructure strategy is? You're not cooking in their kitchen.
>But someone thought they were exceedingly clever and they were going to fix this without any downtime and that's when a small problem became a much larger one
The sentence "and that's when a small problem became a big problem" comes up depressingly frequently in these sorts of post mortems :(. Sometimes sort of feels like, along all the checklists and training and practice and so on, there should also simply be the old Hitchhiker's Guide "Don't Panic!" sprinkled liberally around along with a dabbing of red/orange "...and Don't Be Clever" right after it. We're operating in alternate/direct law here folks, regular assumptions may not hold. Hit the emergency stop button and take a breath.
But of course management and incentive structures play a role in that too.
1. future cryptography attacks that do not exist today
2. Availability of data
3. The legal environment of the data
Encryption is not a panacea that solves every problem
Yes, but your backup DC's can have diesel generators and a few weeks of on-site fuel. It has some quakes - but quake-resistant DC's exist, and SK is big enough to site 3 DC's at the corners of an equilateral triangle with 250km edges. Similar for typhoons. Invading NK armies and nuclear missiles are tougher problems - but having more geography would be of pretty limited use against those.
Until you need them - like with the article here ;) - then the FSB says "only if you do these specific favours for us first...".
The data is never safe thanks to the US Cloud Act.
Has there been any interruption in service?
Whoever made this excuse should be demoted to a journeyman ops engineer. Firing would be too good for them.
We shouldn't rush to judgement.
Ok.
Then it wasn’t a workable design.
The idea of “backup sites” has existed forever. The fact you use the word “cloud” to describe your personal collection of servers doesn’t suddenly mean you don’t need backups in a separate physical site.
If the government mandates its use, it should have a hot site at a minimum. Even without that a physical backup in a separate physical location in case of fire/attack/tsunami/large band of hungry squirrels is a total must-have.
However it was decided that not having that was OK, that decision was negligence.
The project shipped, it's done, they've already moved us onto the next task, no one wants to pay for maintenance anyway.
This has been my experience in 99% of the companies I have worked for in my career, while the engineers that built the bloody thing groan and are well-aware of all the failure modes of the system they've built. No one cares, until it breaks, and hopefully they get the chance to say "I **** told you this was inadequate"
They were still right though: it's absolutely clear without an ounce of doubt that whatever you put on an US cloud is being accessible by the US government, who can also decide to sanction you and deprive you from your ability to access the data yourself.
Not having backups is entirely retarded, but also completely orthogonal.
Also, it's much easier for an intelligence service to get the hand on a 1kB encryption key than on a PB of data: the former is much easier to exfiltrate without being noticed.
And then I don't know why you bring encryption here: pretty much none of the use-case for using a cloud allow for fully encrypted data. (The only one that does is storing encrypted backups on the cloud, but the issue here is that the operator didn't do backups in the first place…)
2. The operative assumption in my statement is that the government does not possess the key. If they do possess it, all bets are off.
3. This thread is about a hypothetical situation in which the Korean government did store backups with a U.S.-based cloud provider, and whether encryption of such backups would provide adequate protection against unwanted intrusion into the data held within.
All bets are off from the start. At some point the CIA managed to get their hands on the French nuclear keys…
> 3. This thread is about a hypothetical situation in which the Korean government did store backups with a U.S.-based cloud provider
This thread is about using US cloud providers, that's it, you are just moving the goalpost.
At least in this case you can track down said officials in their foxholes and give them a good talking-to. Good luck holding AWS/GCP/Azure accountable…
The 3-2-1 backup rule is basic.
The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk.
That’s gross mismanagement.
>Was 1967 a particularly bad winter?
>No, a marvellous winter. We lost no end of embarrassing files.
(While the Credit Lyonnais was investigated in the 90s, both the HQ and the site where they stored their archives were destroyed by fire within a few months)
Encrypt before sending to a third party?
aes128 has been the formal standard for 23 years. The only "foreseeable" event that could challenge it is quantum computing. The likely post quantum replacement is ... aes256, which is already a NIST standard. NIST won't replace aes256 in the foreseeable future.
All that aside, there is no shortage of ciphers. If you are worried about one being broken, chain a few of them together.
And finally, no secret has to last forever. Western governments tend to declassify just about everything after 50 years. After 100 everyone involved is well and truly dead.
The value of said data diminishes with time too. You can totally do an off-site cloud backup with mitigation fallbacks should another country become unfriendly. Hell, shard them such that you need n-of-m backups to reconstruct and host each node in a different jurisdiction.
Not that South Korea couldn't have Samsung's Joyent acquisition handle it.
When the flops required to break an algo exceed the energy available on the planet, items are secure beyond any reasonable doubt.
Jiggling disk heads, modulating fan rates, increasing and decreasing power draw... all are potential information leaks.
Chelsea Manning apparently did it by walking in and out of the facility with a CD marked 'Lady Gaga'. Repeatedly
https://www.theguardian.com/world/2010/nov/28/how-us-embassy...
On the other hand, some popular cryptosystems that were more common in the past have been significantly weakened over the years by mathematical advances. Those were also based on math problems that were believed to be "hard." (They're still very hard actually, but less so than we thought.)
What I'm getting at is that if you have some extremely sensitive data that could still be valuable to an adversary after decades, you know, the type of stuff the government of a developed nation might be holding, you probably shouldn't let it get into the hands of an adversarial nation-state even encrypted.
The current state of encryption is based on math problems many levels harder than the ones that existed a few decades ago. Most vulnerabilities have been due to implementation bugs, and not actual math bugs. Probably the highest profile "actual math" bug is the DUAL_EC_DRBG weakness which was (almost certainly) deliberately inserted by the NSA, and triggered a wave of distrust in not just NIST, but any committee designed encryption standards. This is why people prefer to trust DJB than NIST.
There are enough qualified eyes on most modern open encryption standards that I'd trust them to be as strong as any other assumptions we base huge infrastructure on. Tensile strengths of materials, force of gravity, resistance and heat output of conductive materials, etc, etc.
The material risk to South Korea was almost certainly orders of magnitude greater by not having encrypted backups, than by having encrypted backups, no matter where they were stored (as long as they weren't in the same physical location, obviously).
No you can't. Those aren't hard math problems. They're Universe breaking assertions.
This is not the problem of flight. They're not engineering problems. They're not, "perhaps in the future, we'll figure out..".
Unless our understanding of physics is completely wrong, then None of those things are ever going to happen.
Adding to this...
Most crypto I'm aware of implicitly or explicitly assumes P != NP. That's the right practical assumption, but it's still an major open math problem.
If P = NP then essentially all crypto can be broken with classical (i.e. non-quantum) computers.
I'm not saying that's a practical threat. But it is a "known unknown" that you should assign a probability to in your risk calculus if you're a state thinking about handing over the entirety of your encrypted backups to a potential adversary.
Most of us just want to establish a TLS session or SSH into some machines.
Or how much it is cost to kidnap significant one of key bearer(s)?
I think, it is very reasonable sums for governments of almost any country.
If you have two physically separate places which you could trust key stream, you could use them to backup non-encrypted (or "traditionally" encrypted) data itself, without any OTP.
Btw, you really really need a fresh keystream for each and every backup. You will have as many keystream tapes as you have backup tapes. Re-using the OTP keystream enables a lot of attacks on OTP, e.g. by a simple chosen plaintext an attacker can get the keystream from the backup stream and then decrypt other backup streams with it. XORing similar backup streams also gives the attacker an idea which bits might have changed.
And there is a difference to storing things unencrypted in two locations: If an attacker, like some evil maid, steals a tape in one location, you just immediately destroy its corresponding tape in the other location. That way, the stolen tape will forever be useless to the attacker. Only an attacker that can steal a pair of corresponding tapes in both locations before the theft is noticed could get at the plaintext.
What kind of information might be valuable after so long?
You have to integrate the special software requirements to any cloud storage anyway and hosting a large amount of files isn't an insurmountable technical problem.
If you can provide the minimal requirements like backups, of course.
Which they weren't. And here we are.
That sounds great, as long as nobody makes any mistake. It could be a bug on the RNG which generates the encryption keys. It could be a software or hardware defect which leaks information about the keys (IIRC, some cryptographic system are really sensitive about this, a single bit flip during encryption could make it possible to obtain the private key). It could be someone carelessly leaving the keys in an object storage bucket or source code repository. Or it could be deliberate espionage to obtain the keys.
There was that time when some high profile company's entire Google Cloud account was destroyed. Backups were on Google Cloud too. No off-site backups.
For the budget spent, you’d think they would clone the setup in Busan and sync it daily or something like this in lieu of whatever crazy backup they needed to engineer but couldn’t.
Their 3rd party backups with another provider were crucial to helping them undo the damage from the accidental deletion by GCloud.
GCloud eventually shared a post-mortem [1] about what went down.
0: https://news.ycombinator.com/item?id=40304666
1: https://cloud.google.com/blog/products/infrastructure/detail...
Any sufficiently advanced malice is indistinguishable from stupidity.
I don't think there's anything that can't be attributed to stupidity, so the statement is pointless. Besides, it doesn't really matter naming an action stupidity, when the consequences are indistinguishable from that of malice.
I'm not sure you could realistically explain that as anything. Sometimes ... shit happens.
It's a fallacy to assume that malice is never a form of stupidity/folly. An evil person fails to understand what is truly good because of some kind of folly, e.g. refusing to internally acknowledge the evil consequences of evil actions. There is no clean evil-vs-stupid dichotomy. E.g. is a drunk driver who kill someone with drunk driving stupid or evil? The dangers of drunk driving are well-known, so what about both?
Additionally, we are talking about a system/organization, not a person with a unified will/agenda. There could indeed be an evil person in an organization that wants the organization to do stupid things (not backup properly) in order to be able to hide his misdeeds.
To me, it has no justification. People see malice easily, granted, but others feign ignorance all the time too.
I think a better principle is: Proven and documented testing for competence, making it clear what a persons duties and (liable) responsibilities are, then thereafter treating incompetence and malice the same. Also: any action need to be audited by a second entity who shares blame (to a measured and pre-decided degree) when they fail to do so.
This kind of provision requires enforcement and verification. Thus, a tech spec for the backup procedure. Knowing Germany good enough, I'd say that these tech spec would be detrimental for the actual safety of the backup.
When you live in Germany and are asked to send a FAX (and not a mail, please). Or a digital birth certificate is not accepted until you come with lawyers, or banks not willing to operate with Apple pay, just to name few..
Speculation, yes, but not at all wild
Why wouldn't government mandate storage requirements given the above?
Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.
I know this because I was working on online systems back then.
I also vividly remember 9/11 and the days that followed. We had a satellite dish with multiple receivers (which wasn’t common back then) so had to run a 3rd party Linux box to descramble the single. We watch 24/7 global news on a crappy 5:4 CRT running Windows ME during the attack. Even in the UK, it was a somber and sobering experience.
What used to happen (and still does as far as I know, but I've been out of the networking game for a while now) is you'd get fibre laid between yourself and your ISP. So you're then subject to the latency of their networking stack. And that becomes a huge problem if you want to do any real-time work like DB replicas.
The only way to do automated off-site backups was via overnight snapshots. And you're then running into the bandwidth constraints of the era.
What most businesses ended up doing was tape backups and then physically driving it to another site -- ideally then storing it an fireproof safe. Only the largest companies could afford to push it over fibre.
Yeah because interaction latency matters and legacy/already buried fiber is expensive to rent so you might as well put the facility in range of (not-yet-expensive) 20km optics.
> Copper doesn't work over these kinds of distances without powered switches, which adds latency.
You need a retimer, which adds on the order of 5~20 bits of latency.
> And that becomes a huge problem if you want to do any real-time work like DB replicas.
Almost no application would actually require "zero lost data", so you could get away with streaming a WAL or other form of reliably-replayable transaction log and cap it to an acceptable number of milliseconds of data loss window before applying blocking back pressure. Usually it'd be easy to tolerate enough for the around 3 RTTs you'd really want to keep to cover all usual packet loss without triggering back pressure.
Sure, such a setup isn't cheap, but it's (for a long while now) cheaper than manually fixing the data from the day your primary burned down.
It's "only" poor if you need to restore some files in the middle or want your backup to act as a failover solution to minimise unavailability. But as a last resort solution in case of total destruction, it's pretty much unbeatable cost-wise.
G-Drive was apparently storing less than 1PB of data. That's less than 100 tapes. I guess some files were fairly stable so completely manageable with a dozen of tape drives, delta storage and proper rotation. We are talking of a budget of what 50k$ to 100k$. That's peanuts for a project of this size. Plus the tech has existed for ages and I guess you can find plenty of former data center employees with experience handling this kind of setup. They really have no excuse.
So if real-time synchronization isn't practical, you are then left to do out-of-hours backups and there you start running into bandwidth issues of the time.
Latency is defined by physics (speed of light, through specific conductors or fibres).
Bandwidth is determined by technology, which has advanced markedly in the past 25 years.
Even a quarter century ago, the bandwidth of a station wagon full of tapes was pretty good, even if the latency was high. Physical media transfer to multiple distant points remains a viable back-up strategy should you happen to be bandwidth-constrained in realtime links. The media themselves can be rotated / reused multiple times.
Various cloud service providers have offered such services, effectively a datacentre-in-a-truck, which loads up current data and delivers it, physically, to an off-site or cloud location. A similar current offering from AWS is data transfer terminals: <https://techcrunch.com/2024/12/01/aws-opens-physical-locatio...>. HN discussion: <https://news.ycombinator.com/item?id=42293969>.
Edit to add: from the above HN discussion Amazon retired their "snowmobile" truck-based data transfer service in 2024: <https://www.datacenterdynamics.com/en/news/aws-retires-snowm...>.
I was also specifically responding to the GPs point about latency for DB replication. For backups, one wouldn’t have used live replication back then (nor even now, outside of a few enterprise edge cases).
Snowmobile and its ilk was a hugely expensive service by the way. I’ve spent a fair amount of time migrating broadcasters and movie studios to AWS and it was always cheaper and less risky to upload petabytes from the data centre than it was to ship HDDs to AWS. So after conversations with our AWS account manager and running the numbers, we always ended up just uploading the stuff ourselves.
I’m sure there was a customer who benefited from such a service, but we had petabytes and it wasn’t us. And anyone I worked with who had larger storage requirements didn’t use vanilla S3, so I can’t see how Snowmobile would have worked for them either.
IIRC, multiple IBM mainframes can be setup so they run and are administered as a single system for DR, but there are distance limits.
At a former employer, we used a datacenter in East Brunswick NJ that had mainframes in sysplex with partners in lower manhattan.
But an important factor is, that performance will degrade with every microsecond latency added as the active node for the transaction will have to wait for the acknowledgement of the mirror node (~2*RTT). You can mirror synchronously that distance, but the question is if you can accept the impact.
That's not to say that one shouldn't create a replica in this case. If necessary, synchronize synchronous to a nearby DC and asynchrone to a remote one.
For sure we only know the sad consequences.
I was told right after the bombing, by someone with a large engineering firm (Schlumberger or Bechtel), that the bombers could have brought the building down had they done it right.
That's like storing lifeboats in the bilge section of the ship, so they won't get damaged by storms.
That being said, I can likely guess where this ends up going:
* Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite the program in question (G-DRIVE) existing since 2017 in some capacity.
* Nobody in government will question sufficiently what technical reason is/was given to justify the lack of backups and why that was never addressed, why the system went live with such a glaring oversight, etc, because that would mean holding the actual culprits accountable for mismanagement
* Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations
* The major cloud providers will likely win several contracts for “temporary services” that in actuality strip the sovereignty the government had in managing its own system, even if they did so poorly
* Other countries will use this to justify outsourcing their own sovereign infrastructure to private enterprise
This whole situation sucks ass because nothing good is likely to come of this, other than maybe a handful of smart teams lead by equally competent managers using this to get better backup resources for themselves.
Practically speaking, I think a lot of what is offered by Microsoft, Google, and the other big companies that are selling into this space is vastly overpriced and way too full of lock-in, taking this stuff in-house without sufficient knowhow and maturity is even more foolish.
It's like not hiring professional truck drivers, but instead of at least people who can basically drive a truck, hiring someone who doesn't even know how to drive a car.
Now, sure, there is AWS Backup and Microsoft 365 Backup. Nevertheless, those are backups in the same logical environment.
If you’re a central government, you still need to be maintaining an independent and basically functional backup that you control.
I own a small business of three people and we still run Veeam for 365 and keep backups in multiple clouds, multiple regions, and on disparate hardware.
They might (MIGHT) get fired from their government jobs, but I'll bet they land in consulting shops because of their knowledge of how the government's IT teams operate.
I'll also bet the internal audit team slides out of this completely unscathed.
They really, really shouldn't. However, if they were shouted down by management (an unfortunately common experience) then it's on management.
The trouble is that you can either be effective at internal audit or popular, and lots of CAE's choose the wrong option (but then, people like having jobs so I dunno).
Also, internal audit aren't supposed to be the only audit, they are effectively pre-audit prep for external audit. And the first thing an external auditor should do - ask them probing questions about their systems and process.
It's negligence all the way, not just with this G-Drive designers, but with customers as well.
This isn't to preclude major ignorance in terms of those in the technology departments themselves. Having worked in/around govt projects a number of times, you will see some "interesting" opinions and positions. Especially around (mis)understanding security.
The real question then is more fundamental.
For anyone else who's confused, G-Drive means Government Drive, not Google Drive.
Having worked a great deal inside of aws on these things aws provides literally every conceivable level of customer managed security down to customer owned and keyed datacenters operated by aws, with master key HSMs owned, purchased by the customer, with customer managed key hierarchies at all levels and detailed audit logs of everything done by everything including aws itself. The security assurance of aws is far and away beyond what even the most sophisticated state actor infrastructure does and is more modern to boot - because it’s profit incentive drives that.
Most likely this was not about national security than about nationalism. They’re easily confused but that’s fallacious. And they earned the dividends of fallacious thinking.
Not using a cloud provider is asinine. You can use layered encryption so the expected lifetime of the cryptography is beyond the value of the data...and the US government themselves store data on all 3 of them, to my knowledge.
I say US because the only other major cloud providers I know of are in China, and they do have a vested interest in South Korean data, presumably for NK.
Can you articulate what particular advantages the US would be pursuing by stealing SK secret data (assuming it was not protected sufficiently on AWS/GCP to prevent this, and assuming that platform security features have to be defeated to extract this data—this is a lot of risk from the US's side, to go after this data, if they are found out in this hypothetical, I might add, so "they would steal whatever just to have it" is doubtful to me).
The US also has a secret spy facility in Pine Gap that is believed to (among other things) spy on Australian communications, again despite both countries being very close allies. No Australians know what happens at Pine Gap, so maybe they just sit around knitting all day, but it seems somewhat unlikely.
[1]: https://www.theguardian.com/us-news/2015/jul/08/nsa-tapped-g...
Why do you think USA wouldn't lie, cheat and spy on someone if it had a benefit in it?
Or ZFS or DRBD or whatever homegrown or equivalent non-proprietart alternative is available these days and you prefer.
They absolutely cannot be trusted, especially sensitive govt. data. Can you imagine the US state department getting their hands on compromising data on Korean politicians?
Its like handing over the govt. to US interests wholesale.
That they did not choose to keep the backup, and then another, at different physical locations is a valuable lesson, and must lead to even better design the next time.
But the solution is not to keep it in US hands.
Obviously no person in a lawmaking position will ever have the patience or foresight to learn about this, but the fact they won't even try is all the more infuriating.
It sounds technologically illiterate to you because when people say "we can't safely use a foreign cloud" you think they're saying "to store data" and everyone else is thinking at the very least "to store and process data".
Sure, they could have used a cloud provider for encrypted backups, but if they knew how to do proper backups, they wouldn't be in this mess to begin with.
Ha! "Did not allow" my ass. Let me translate:
> We didn't feel like backing anything up or insisting on that functionality.
This has nothing to do with commercial clouds. Commercial clouds are just datacenters. They could pick one commercial cloud data center and not do much more to mirror or backup in different regions. I understand some of the services have inherent backups.
there are many failure points here, not paying Amazon/Google/Microsoft is hardly the main point.
We use selfhosting, but we also test our fire suppression system every year, we have two different DCs, and we use S3 backups out of town.
Whoever runs that IT department needs to be run out of the country.
You have a backup copy of your file, in the same folder? That helps for some "oops" moments, but nothing else.
You have a whole backup DRIVE on your desktop? That's better. Physical failure of the primary device is no longer a danger. But your house could burn down.
You have an alternate backup stored at a trusted friend's house across the street? Better! But what if a major natural disaster happens?
True life, 30+ years ago when I worked for TeleCheck, data was their lifeblood. Every week a systems operator went to Denver, the alternate site, with a briefcase full of backup tapes. TeleCheck was based in Houston, so a major hurricane could've been a major problem.
[0]https://www.theregister.com/2025/07/25/microsoft_admits_it_c...
Reality hits later when budget cuts/constrained salaries prevent the maintenance of a competent team. Or the proposed backup system is deemed as excessively risk averse and the money can’t be spared.
They can't. The trump admin sanctioning the international criminal court and Microsoft blocking them from all services as a result are proof of why.
There may be a point in time it made sense but high resolution detailed satellite imagery is plenty accessible and someone could put a road and basically planning structure atop it, especially a foreign nation wishing to invade or whatever they’re protecting against.
Some argument may be made that it would be a heavy lift for North Korea but I don’t buy it, incredibly inconvenient for tourists for no obvious reason.
If you take a moment to think about it, what's weird is that so many countries have simply resorted to relying on Google Maps for everyday mapping and navigation needs. This has become such a necessity nowadays that relying on a foreign private corporation for it sounds like a liability.
What’s odd (to me) is trying to regulate other groups from generating maps of your nation when you have no jurisdiction over them. That’s akin to the US telling all South Korean governments they can’t create maps of the US unless they operate under heavy supervision or something of that nature.
It’s impractical, largely unenforceable, and any nation probably has independent mapping of foreign nations, especially their adversaries, should they need them for conflicts, regardless of what some nation wants to oppose over them in terms of restrictions. I guarantee the US government has highly detailed maps of Korea.
So who exactly are these regulations protecting? In this case they’re just protecting private mapping groups that reside in their country against competition.
What made you think of them as bad? Could you be more specific? I use them almost daily and I find them very good.
I heard similar complaints from friends that came to visit. But they were using the English version of the apps, which, when I tested, were indeed harder to use, but never a miss for me when I helped them. OTOH, I always find my destinations within the first three options when I search in Korean. So maybe it's subpar internationlization.
> They lack a lot of polish. [...] some interactions are janky
I see. I guess I wouldn't know. It's not janky for me, and I think that I am so used to it that when I need to use Google Maps, or any other, I feel a bit frustrated by the unfamiliar interface that I start wishing I could be using Kakao or Naver Maps instead.
Maybe more about my unfamiliarity with the Korean address format than anything else tbh.
Some things about Naver I kind of miss from Apple/Google maps, but international software in general feels much more user friendly and better UX than Korean software.
I believe performance wise it was also pretty sluggish from what I remember. I’m by no means saying it was unusable, it got me through somewhat functionally but with a lot of extra effort on my behalf. I also had an international data plan and wasn’t able to see if I could precache the map set vs streaming it as needed over wireless.
I often like to look at restaurants, menus, prices, reviews as well to scope out a place quickly before going there. That process was also tedious (to be fair it could be that I’m not familiar with the UI).
The question is why did I have to use Naver or Kakao in the first place. I’d rather just use the system I already enjoy and am quite proficient with using it, not be forced to play with some new app that I need useful information from for some unclear reason.
> A senior government official overseeing recovery efforts for South Korea's national network crisis has reportedly died by suicide in Sejong.
Even in cases it is executed by themselves, shame won't be the primary motivation.
More often than not the suicide covers a whole organization's dirty laundry. You'll have people drunk and driving their cars over cliffs [0], low profile actors ending their life as shit hits the fan [0] etc.
Then some on the lower rank might still end their life to spare their family financially (insurance money) or because they're just so done with it all, which I'd put more on depression than anything.
Us putting it on shame is IMHO looking at it through rose colored glasses and masking the dirtier reality to make it romantic.
[0] https://bunshun.jp/articles/-/76130
[1] https://www.tsukubabank.co.jp/cms/article/a9362e73a19dc0efcf...
Shame is a powerful social force throughout the society, but we're talking about systematic screwings more often than not backed by political corruption (letting incompetent entities deal with gov contract on basis of political money and other favors) or straight fraud.
I'm not sure Yoon Suk Yeol had any shame
It’s pretty crazy over there, Lee’s probably safe right now just because his party’s the majority. But it also sounds like they’ve been abusing the impeachment process against the minority party.
> My wife, who’s from Korea
> 13 days later his temporary successor Han was also impeached
- Han Duck-soo: Acting president for 13 days. Impeached for refusing to investigate Yoon Suk Yeol and Kim Keon Hee (Yoon's wife).
- There were 192 votes against him and 108 members *abstained* from voting. This meant that they failed to form a quorum. *This vote was strictly party lines*
- They ruled that they only need 50% approval because Han was the Prime Minister. *President needs 2/3rds btw*
- Choi Sang-mok: was the acting PM for those 13 days. But only serves for 87!
- 24 March SK's (equivalent to) supreme court overrules Han's impeachment 7-1, and Han once again becomes the acting president.
> the DPK has been impeaching everyone they disagree with.
Before Yoon was Moon Jae-in (DPK), who the GNP tried to impeach in 2019. (Hwang Kyo-ahn was acting after Park's impeachment, who preceded Moon).
Before Park was Lee Myung-bak (GNP). He got 15 years in prison. In 2022 Yoon gave him a pardon.
Before Lee was Roh Moo-hyun (Liberal party) (Goh Kun was in between because...) but was impeached (193 to 2) in 2004 and his supporters were literally fighting people in the assembly. Month later supreme court overturned impeachment. After he left presidency people around him started getting sentenced. In 2009 he threw himself off a cliff as investigations were following him too.
Since the 60's they've had a president exiled, a coup, and even an assassination. It's fucking wild!
And don't get started on chaebols...[3]
[0] https://en.wikipedia.org/wiki/List_of_presidents_of_South_Ko...
[1] https://en.wikipedia.org/wiki/Impeachment_of_Han_Duck-soo
The plane's aft pressure bulkhead catastrophically exploded, causing total decompression at the high altitude, severing all four of the massive plane's hydraulic stabilizer systems and entirely tearing away its vertical stabilizer.
With these the 747 basically became uncontrollable and minutes later, despite tremendously heroic efforts by the pilots to turn back and crash land it with some modicum of survivability for themselves and the passengers, the flight slammed into a mountain close to Tokyo, killing hundreds.
The resulting investigation showed that the failed bulkhead had burst open due to faulty repair welding several years before. The two technicians most responsible for clearing that particular shoddy repair both committed suicide soon after the crash tragedy. One of them even left a note specifically stating "With my death I atone". (paraphrasing from memory here)
I can't even begin to imagine a modern Boeing executive or senior staffer doing the same.
Same couldn't be said for Japanese military officials after the tragedy though, so who knows about cultural tendencies:
Right after the crash, helicopters were making ready to fly to the scene (it was night by this point) and a nearby U.S military helicopter squadron also even offered to fly in immediately. The local JSDF administration however stood all these requests down until the following morning, on the claim that such a tremendous crash must not have left anyone alive, so why hurry?
As it turned out, quite a number of people had incredibly survived, and slowly died during the night from exposure to cold and their wounds, according to testimony from the four who did survive to be rescued, and doctors who later conducted postmortems on the bodies.
On the other hand you have cases like the MV Wewol ferry disaster, also in South Korea, in which well over 250 passengers died horribly. Most of them were just kids, high school students on a trip. The causes leading up to the tragedy, the accident management by the crew itself and the subsequent rescue, body retrieval and investigation, were absolutely riddled with negligence, incompetence, bad management and all kinds of blame shifting.
The owner of the ferry company itself had an arrest warrant issued for him, then fled and only later was found in a field dead and presumed to have committed suicide.
Underlying all this is that even these apparent cultural ideas of committing suicide to atone for the shame of some gigantic mistake don't seem to prevent people from actually making these kinds of mistakes or doing things more responsibly in the first place.
Seriously, I work here as an IT guy and I can't stop wondering how they could become so advance in other areas and stay so backwards in anything software-related except videogames.
That's why they are good for industrial processes where they can iterate and improve in small, incremental steps, but terrible for software projects full of uncertainties.
Recently I have been thinking about whether we actually need governments, nation states and all of the hubris that goes with it such as new media. Technically this means 'anarchism' with everyone running riot and chaos. But, that is just a big fear, however, the more I think through the 'no government' idea, the less ludicrous it sounds. Much can be devolved to local government, and so much else isn't really needed.
South Korea's government have kind-of deleted themselves and my suspicion is that, although a bad day for some, life will go on and everything will be just fine. In time some might even be relieved that they don't have this vast data store any more. Regardless, it is an interesting story regarding my thoughts regarding the benefits of no government.
Maybe it was just convenient to have the possibility of losing everything.
>However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost.
I think they decided that their storage was too slow to allow backups?
Seems hard to believe that they couldn't manage any backups... other sources said they had around 900TB of storage. An LTO-9 tape drive holds ~20TB uncompressed, so they could have backed up the entire system with 45 tapes. At 300MB/sec with a single drive, it would take them a month to complete a full backup, so seems like even a slow storage system should be able to keep up with that rate. They'd have a backup that's always a month out of date, but that seems better than no backup at all.
If your storage performance is low then you don't need fat pipes to your external provider either.
They either built this too quickly or there was too much industry corruption perverting the process and the government bought an off the shelf solution that was inadequate for their actual needs.
LTO-9 ~$92/tape in bulk. A 4 drive library with 80 drive capacity costs ~$40k* and can sustain about 1 Gbps. It also needs someone to barcode, inventory, and swap tapes once a week and an off-site vaulting provider like Iron Mountain. That's another $100k/year. Also, that tape library will need to be replaced every 4-7 years, so say 6 years. And those tapes wear out over X uses and sometimes go bad too. It might also require buying a server and/or backup/DR software too. Furthermore, a fire-rated data safe is recommended for about 1-2 weeks' worth of backups and spare media. Budget at least $200k/year for off-site tape backups for a minimal operation. (Let me tell you about the pains of self-destructing SSL2020 AIT-2 Sony drives.)
If backups for other critical services and this were combined, it would probably be cheaper to scale this kind of service rather reinventing the wheel for just one use-case in one department. That would allow for possibly multiple types of optimizations like network-based backups to nearline storage to then be streamed more directly to tape and using many more tape drives, possibly a tape silo robot(s) and perhaps split into 2-3 backup locations obviating the need for off-site vaulting.
Furthermore, it might be simpler, although more expensive, to operate another hot-/warm-site for backups and temporary business continuity restoration using a pile of HDDs and a network connection that's probably faster than that tape library. (Use backups, not replication because replication of errors to other sites is fail.)
Or the easiest option is to use one or more cloud vendors for even more $$$ (build vs. buy tradeoff).
* Traditionally (~20 years ago), enterprise "retail" prices of gear was sold at around 100% markup allowing for up to around 50% discount when negotiated in large orders. Enterprise gear also had a lifecycle of around 4.5 years while it still might technically work, there wouldn't be vendor support or replacements for them, and so enterprise customers are locked into perpetual planned obsolescence consumption cycles.
At least now they see the true cost of not having any off site backups. It's a lot more than $0.66 per user.
A typical 2005 era drive from two decades ago might have been 0.5 TB with a throughput of 70 MB/s for a full-drive transfer time (FDTT) of about 2 hours. A modern 32 TB drive is 64x bigger but has a throughput of only 270 MB/s which is less than 4x higher. Hence the FDDT is 33 hours!
This is the optimal scenario, things get worse in modern high-density disk arrays that may have 50 drives in a single enclosure with as little as 8-32 Gbps (1 GB/sec to 4 GB/sec) of effective bandwidth. That can push FDDT times out to many days or even weeks.
I've seen storage arrays where the drive trays were daisy chained, which meant that while the individual ports were fast, the bandwidth per drive would drop precipitously as capacity was expanded.
It's a very easy mistake to just keep buying more drives, plugging them in, and never going back to the whiteboard to rethink the HA/DR architecture and timings. The team doing this kind of BAU upgrade/maintenance is not the team that designed the thing originally!
Koreans are weird, for example they will rather eat contractual penalty than report problems to the boss.
OVH's massive fire a couple of years ago in one of the most modern DC's at the time was a prime example of just how wrong it can go.
I guess not.
There are some in this threading pointing out that this would be handled by cloud providers. That bad - you can't hope for transparent backup, you need to actively have a discipline over it.
My fear is that our profession has become very amateurish over the past decade and a lot of people are vulnerable to this kind of threat.
To be some context on Korea IT scene, you get pretty good pay and benefits if you work for a big product company, but will be treated like dogshit inside subcontracting hell if you work anywhere else.
[1] https://www.hani.co.kr/arti/society/society_general/1222145....
I noticed that someone had daisy-chained petabytes of disk through relatively slow ports and hadn’t enabled the site-to-site replication that they had the hardware for! They had the dark fibre, the long-range SFPs, they even licensed the HA replication feature from the storage array vendor.
I figured that in a disaster just like this, the time to recover from the tape backups — assuming they were rotated off site, which might not have been the case — would have been six to eight weeks minimum, during which a huge chunk of the government would be down. A war might be less disruptive.
I raised a stink and insisted that the drives be rearranged with higher bandwidth and that the site-to-site replication be turned on.
I was a screamed at. I was called unprofessional. “Not a team player.” Several people tried to get me fired.
At one point this all culminated in a meeting where the lead architect stood up in front of dozens of people and calmly told everyone to understand one critical aspect of his beautiful design: No hardware replication!!!
(Remember: they had paid for hardware replication! The kit had arrived! The licenses were installed!)
I was younger and brave enough to put my hand up and ask “why?”
The screeched reply was: The on-prem architecture must be “cloud compatible”. To clarify: He thought that hardware-replicated data couldn’t be replicated to the cloud in the future.
This was some of the dumbest shit I had ever heard in my life, but there you go: decision made.
This. This is how disasters like the one in South Korea happen.
In private organisations you get competent shouty people at the top insisting on a job done right. In government you get incompetent shouty people insisting that the job gets done wrong.
Great post and story but this conclusion is questionable. These kinds of incompetences or misaligned incentives absolutely happen in private organisations as well.
There’s a sort-of “gradient descent” optimisation in private organisations, established by the profit motive and the competitors nipping at their heels. There’s no such gradient in government, it’s just “flat”. Promotions hence have a much weaker correlation with competence and a stronger correlation with nepotism, political skill, and willingness to participate in corruption.
I’ve worked with may senior leaders in all kinds of organisations, but only in government will you find someone who is functionally illiterate and innumerate in a position of significant power.
Obviously this is just a statistical bias, so there’s overlap and outliers. Large, established monopoly corporations can be nigh indistinguishable from a government agency.
Is this going to have a real impact in the near term? What kind of data are we’re talking about being permanently lost?
That said, people do make bad calls, and this would be an epically bad one, if they really don't have any form of backup.
[1] These days creating such a facility for archiving an exabyte of essentially write mostly data are quite feasible. See this paper from nearly 20 years ago: https://research.ibm.com/publications/ibm-intelligent-bricks...
It's funny that you mention that...
Many years ago I was Unix sysadmin responsible for backups and that is exactly what we did. Once a week we rotated the backup tapes taking the oldest out of the fire safe and putting the newest in. The fire safe was in a different building.
I thought that this was quite a normal practice.
https://www.thestar.com.my/aseanplus/aseanplus-news/2025/10/...
But the problem is they sacrificed "Availability" in pursuit of security and privacy. Losing your data to natural and man-made disasters is one of the biggest risks facing any storage infrastructure. Any system that cannot protect your data against those should never be deployed.
"The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups."
This is not a surprise to them. They had knowingly accepted the risk of infrastructure being destroyed by natural and man-made disasters. I mean, WTF!
South Korea isn't some poor backwater, they have tech companies and expertise, that they were "unable" to do backups was an intentional choice.
:facepalm:
> This article was originally written in Korean and translated by a bilingual reporter with the help of generative AI tools. It was then edited by a native English-speaking editor. All AI-assisted translations are reviewed and refined by our newsroom.
https://www.bloodinthemachine.com/p/ai-killed-my-job-transla...
First it gave a "verbatim, literal English translation" and then asked me if I would like "a version that reads naturally in English (but still faithful to the tone and details), or do you want to keep this purely literal one?"
Honestly, the English translation was perfect. I know Spanish, I knew the topic of the article and had read about it in the NYTimes and other English sources, and I am a native English speaker. It's sad, but you can't put the toothpaste back in the tube. LLMs can translate well, and the article saying otherwise is just not being intellectually honest.
I want to know how it holds up translating Spanish to Farsi, for example.
A classic issue is dealing with things like wordplay. Good bilingual editors might be able to get across the intended meaning in other ways, but I highly doubt translation software is capable of even recognizing it.
Funnily enough, I'd say it reads like most of my American friends here in Spain - the best way I can put it is, it's fluid spanish from a brain that is working natively in English and translating on the fly, rather than a mind thinking in Spanish.
This is obvious to me because I speak both languages, so I can trace back in my mind the original, native English phrase that resulted in a specific weird spanish expression. a Spanish monolingual can probably only tell that it doesn't sound native.
The important point though, is that there is no significant loss of meaning other than the text being annoying to read. it won't work for literature but it's perfectly serviceable for pragmatic needs.
(Whether machine translation uses LLMs or not doesn’t seem especially relevant to the workflow.)
The AI hype train is bs, but there're real and concrete uses for it if you don't expect it to become a super-intelligence.
When you consider that there are real and concrete uses for it across a wide variety of domains, the hype starts to make more sense.
Obviously Sam “we’ll build a Dyson sphere with it” is off in hype lala land somewhere while he tries to raise a trillion dollars to burn through fossil fuels as fast as possible, but that’s a kind of symptom of the real underlying capabilities and promise here.
> The AI hype train is bs, but there're real and concrete uses for it if you don't expect it to become a super-intelligence.
While LLM usage is just spinning up in other areas, for translation they have been doing this job well for over 5 years now.
GNMT used seq2seq with attention to do translations. GNMT plus some RNN and attention lead to transformers, and here we are today.
Oh?
phew I'm relieved you're okay with people using modern tools to get their job done
The final note that all AI-assisted translations are reviewed by the newsroom is also interesting. If they are going to take the time to review it and have enough experience in both languages to verify the translation, why use the LLM for it at all?
I've done my fair share of translating as a bilingual person and having an LLM to do a first pass at translation saves TON of time. I don't "need" LLM, but it's definitely a helpful tool.
People generally read (and make minor edits if necessary) much faster than they can write.
Two different editors.
But as others mentioned, this is helpful even for the same editor to do.
This will give the reporter more time to work on more articles, and we as a foreigner to Korea, getting more authentic Korean news that is reviewed by Korean and not be Google Translate.
> If the reporter needs an LLM to help translate they could easily be missing subtle mistranslations.
A New Gaza Rage Machine–With Polish Origins - https://news.ycombinator.com/item?id=45453533
No offsite backups. No onsite backups. No usb drives laying around unsecure in a closet. Nothing.
What?
Top fucking kek. What were they expecting to happen? Like, really? What were they thinking?
I know that I have had to keep informal copies of valuable systems because the real source of truth is continually patched,offline,churn,whatever.
technically, it was the supervising technical director.
The only reason this happened (I don't think "working from home" was very common in 1999) was because she just had a baby! I love this story because it feels like good karma – management providing special accommodations for a new mom saves the show.
On the other hand, I hope a few boots on the ground get to use this as a chance to toss decades of bad technical debt. "Why are we still running that 2011 Oracle database version?".
The very first "computer guy" job I had starting in about 1990/1991, my mentor gave me a piece of advice that I remember to this day: "Your job is to make sure the backups are working; everything else is gravy."
While I worked in that job, we outgrew the tape backup system we were using, so I started replicating critical data between our two sites (using 14400 bps Shiva NetModems), and every month I'd write a memo requesting a working backup system and explaining the situation. Business was too cheap to buy it.
We had a hard drive failure on one of our servers, I requested permission to invalidate the drive's warranty because I was pretty sure it was a bad bearing; I got it working for a few weeks by opening the case and spinning the platter with my finger to get it started. I made sure a manager was present so that they could understand how wack the situation was- they bought me a new drive but not the extras that I asked for, in order to mirror.
After I left that job, a friend of mine called me a month later and told me that they had a server failure and were trying to blame the lack of backups on me; fortunately my successor found my stack of memos.
Did anything change? No.
Those, ironically, were backed up
In other words.. there was no point in the extra security of being onsite AND the risks of being onsite single failure point destroyed any evidence.
Pretty much what I'd expect tbh, but no remote backup is insane.
> The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets.
No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place.
Two of those would be less than the price of a new Hyundai Grandeur car.
> “It’s daunting as eight years’ worth of work materials have completely disappeared.”
So they're clocking in at around 100TB/year or 280GB a day. It's respectable, but not crazy. It's about 12GB/hr, doable with professional, server level hardware with backup moved over dedicated fiber to an offsite location. Multiply the price 10x and you can SSD the entire thing.
Even with data sovereignty consideration demanding an entirely 100% home grown solution rather than turning to AWS or Azure, there's no excuse. But it's not like the cloud providers don't already have CSAP certification and local, in country, sovereign clouds [1] with multiple geographic locations in country [2]
South Korea is full of granite mountains, maybe its time the government converts one into an offsite, redundant backup vault?
1 - https://erp.today/south-korea-microsoft-azure-first-hypersca...
2 - https://learn.microsoft.com/en-us/azure/reliability/regions-...
If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the data set on them, each with primary secondary backup. Enterprise flash has been under a dollar a gigabyte for over a decade, so that’s under $1.7M per copy, amortized over five years. That’s $700K per year, and one of the four copies (at 3-4 sites) could be the primary store.
(I can’t be bothered to look up current prices, but moore’s law says there have been six capacity doublings since then, and it still applies to flash and networking, so divide my estimate by 2^6 — so, ten-ish grand per year, with zero full time babysitters required).
There's a high chance i'm missing something though, where did you read about a G-Drive backup system?
Sometimes I wonder why I still try and save disk space :-/
If you use such consumer drives, I strongly suggest to make occasional offsite backups of large mostly static files (movies for most people I guess), and frequent backups of more volatile directories to an offsite place, maybe encrypted in the cloud.
Backup operations are often complex and difficult - but then again it's been worked on for decades and rigorous protocols exist which can and should be adopted.
"However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained" ... "the G-Drive’s structure did not allow for external backups."
Clearly [in]competence was the single factor here.
This is what happens when you come up with all kind of reasons to do something yourself, which you are not qualified to do, rather than simply paying a vendor to do it for you.
It quickly becomes much less so if you satisfy yourself with very crude methods.
Sure that would be an imperfect backup in many ways but any imperfect backup is always infinitely better than no backup at all.
[1]: https://www.computerweekly.com/news/366629871/Microsoft-refu... [2]: https://lcrdc.co.uk/industry-news/microsoft-admits-no-guaran...
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
First, (as you guys have seen) South Korea's IT security track record is not great. Many high-profile commercial sites have been hacked. If a government site was hacked by North Korea, it won't be the first, and while it would be another source of political bickering and finger-pointing, it's likely to blow over in a month.
In fact, given that SK's president Lee started his term in June after his predecessor Yoon's disastrous attempt at overthrowing the constitution, Lee could easily frame this as a proof of the Yoon admin's incompetence.
But deliberately setting fire on a government data center? Now that's a career ending move. If that's found out, someone's going to prison for the rest of their life. Someone should be really desperate to attempt that kind of thing. But what thing could be so horrible that they would rather risk everything to burn the evidence? Merely "we got hacked by North Korea" doesn't cut it.
Which brings us to the method. A bunch of old lithium batteries, overdue for replacement, and predictably the job was sold to the lowest bidder - and the police knows the identity of everyone involved in the job and is questioning them as we speak.
So if you are the evil perpetrator, either you bribed one of the lowest wage workers to start a fire (and the guy is being questioned right now), or you just hoped one of the age-old batteries would randomly start fire. Neither sounds like a good plan.
Which brings us to the question "Why do people consider that plausible?" And that's a doozy.
Did I mention that President Yoon almost started a coup and got kicked out? Among the countless stupid things he did, he somehow got hooked up on election conspiracy theories that say that South Korea's election commission was infiltrated by Chinese spies (along with major political parties, newspapers, courts, schools, and everything) and they cooked the numbers to make the (then incumbent) People's Power Party to lose congressional election of 2024.
Of course, the theory breaks down the moment you look close. If Chinese spies had that much power, how come they let Yoon win his own election in 2022? Never mind that South Korea uses paper ballots and every ballot and every voting place is counted under the watch of representatives from multiple parties. To change numbers in one counting place, you'll have to bribe at least a dozen people. Good luck doing that at a national scale.
But somehow that doesn't deter those devoted conspiracy theorists, and now there are millions of idiots in South Korea who shout "Yoon Again" and believe our lord savior Trump will come to Korea any day soon, smite Chinese spy Lee and communist Democratic Party from their seats, and restore Yoon at his rightful place at the presidential office.
(Really, South Korea was fortunate that Yoon had the charisma of a wet sack of potatoes. If he were half as good as Trump, who knows what would have happened ...)
So, if you listen to the news from South Korea, and somehow there's a lot of noise about Chinese masterminds controlling everything in South Korea ... well now you know what's going on.
> "The Veritable Records of the Joseon Dynasty, sometimes called sillok (실록) for short, are state-compiled and published records, documenting the reigns of the kings of the Joseon dynasty in Korea. Kept from 1392 to 1865, they comprise 1,893 volumes and are thought to be the longest continual documentation of a single dynasty in the world."
> "Beginning in 1445, they began creating three additional copies of the records, which they distributed at various locations around Korea for safekeeping."
https://en.wikipedia.org/wiki/Veritable_Records_of_the_Joseo...
After the Japanese and Qing invasions of Japan, King Hyeonjong (1659–1675) started a project to collect calligraphy works written by preceding Joseon kings and carve them into stone.
It's somewhat surprising that these values didn't continue to persist in the Korean government.
At a minimum, they could've stored the important bits like financial transactions, personnel/HR records, and asset inventory database backups to Tarsnap [0] and shoved the rest in encrypted tar backups to a couple of different providers like S3 Glacier and/or Box.
Business impact analysis (BIA) is a straightforward way to assessing risks of probability of event * cost to recover from event = approximate budget for spending on mitigation.
And, PSA: test your backups and DR/BCP runbooks periodically!
I can share an anecdote how slow tech adoption is in Korea. It is not exactly about tech in public section but in private companies. I assume public section has slower adoption rate than private ones in general.
Just about a year ago I had a couple of projects with insurance companies. I won't name them but they are the largest ones whose headquarters you can find in the very center of Seoul. They often called me in because I was setting up on-premise servers for the projects. Not to mention that it was hard to understand their choices of database architecture to plug it into the server I was setting up, their data team seemed just incompetent, not knowing what they were doing.
The wildest thing I found was that most office workers seemed to be using windows 2000 to run their proprietary software. To be fair, I like software UIs with a lot of buttons and windows from that era. But alas, I didn't want to imagine myself connecting that legacy software to my then project service. It didn't go that far in the end.
Things South Korea is bad at producing: Software.
Not too bad overall.
https://oec.world/en/visualize/tree_map/hs92/export/usa/all/...
Side note: Why is there so much fact-free anti-US sentiment on HN?
People start believing we can't do anything at all.
Look at basically any domain: whoever's in the lead, gets the most hate.
The US has the world's biggest economy and military, the most cultural power, the biggest tech industry, etc. The hate is inevitable. That's not to say that the US doesn't have plenty of very real problems -- obviously it does -- but it's just easier to dunk on the leader, which also means more empty, vacuous criticism.
Firstly, it's the US government themselves saying there are imbalances and therefore they have to add tariffs on imports from almost every country. It's the US government who spreads hate towards most other countries, not the other way around.
Secondly, could it be because people living in the US seem to not notice (or don't want to believe) the US is turning into a dictatorship and the rest of the world does. People don't like the new values of the USA, they liked the old values. If it continues like this, it's game over for the USA.
Also: music and TV shows.
> Things South Korea is bad at producing: Software.
Also: babies.
I guess it's not all tech, but at least in telecoms I thought they were very quick to adopt new tech? 2nd in the world to commercially deploy 3G W-CDMA, world first LTE-Advanced [1], "first fairly substantial deployments" of 5G [2]. 90% of broadband via fibre (used to be #1 amongst OECD countries for some time, now it's only just #2).
[1] https://en.wikipedia.org/wiki/SK_Telecom#History
[2] https://en.wikipedia.org/wiki/5G#Deployment
[3] https://www.oecd.org/en/topics/sub-issues/broadband-statisti... -> Percentage of fibre connections in total broadband (June 2024) spreadsheet link
I applaud them for honouring their obligation to keep such data private. And encourage them to work on their backup procedures while continuing to honour that obligation.
It's possible that this is all just a coincidence, but the possibility that North Korea is trying to cover their tracks is there.
And the cost of everything is PAIN to us.
If our building burned down we would lose data, but only the data we are Ok with losing in a fire.
I'd love to know the real reason. It's not some useless tech... it's politics, surely.
That should be classified as willful sabotage. Someone looked at the cost line for having backups in another location and slashed that budget to make numbers look good.
I’m almost sure that the system had some sort of local replication and versioning that was enough to deal with occasional deletions, rollbacks, and single non-widespread hardware failures, so only the very catastrophic scenario of losing all servers at the same time (that for sure wouldn’t happen anytime soon) was uncovered.
Apparently I plan for the rather rare catastrophes, and not those customers care about day to day.
What, no backup(s) set up? Hmmm, possibly. But, they're'd be a paper trail.
Imagine all the scrabbling going on right now - people desperately starting to cover their arses. But chances are, what they need has just burnt down, with no backups.
Seriously, "no backups available" for a national government's main cloud storage? That’s not a simple IT oversight; that’s an epic, unforgivable institutional mistake.
It completely exposes the biggest fear everyone in tech has: putting all the eggs in one big physical basket.
I mean, we all know the rule: if it exists in only one place, it doesn't really exist. If your phone breaks, you still have your photos on a different server, right? Now imagine that basic, common-sense rule being ignored for a country’s central data.
The fire itself is a disaster, but the real catastrophe is the planning failure. They spent millions on a complex cloud system, but they skipped the $5 solution: replicating the data somewhere else—like in a different city, or even just another building across town.
Years of official work, policy documents, and data—just gone, literally up in smoke, because they violated the most fundamental rule of data management. This is a massive, expensive, painful lesson for every government and company in the world: your fancy cloud setup is worthless if your disaster recovery plan is just "hope the building doesn't burn down." It’s an infrastructure nightmare.
Thus, the not-so-easy-but-arguably-better solution would be to self-host an open source S3-compatible object storage solution.
Are there any good open source alternatives to S3?
One dimension is two backups can be close in space (ie, physically close, as happened here). Ergo backups must be physically separated.
You've heard RAID can't be a backup? Well it sort of can, and the two drives can be physically separated in space. But they are connected in another dimension - time, as in they reflect the data at the same instant in time. So if you have a software failure that corrupts all copies, your backups are toast as you can't go back to a previous point in time to recover.
Another dimension is administrative control. Google Drive for example will backup your stuff, and separate it in space and time. But they are connected by who controls them. If you don't pay the bill or piss Google off, you've lost all your backups. I swear every week I see a headline saying someone lost their data this way.
Then backups can be all connected to you via one internet link, or connected to one electrical grid, or even one country that goes rogue. All of those are what I called dimensions, that you have to ensure your backups are held at a different location in each dimension.
Sorry, that didn't answer your question. The answer no. It's always possible all copies could be wiped out at the same time. You are always relying on luck, and perhaps prayer if you think that helps your luck.
Amazing
Just so we can all visualise this in an understandable way, if laid end-to-end how many times round the world would the A4 sheets go?
And what is their total area in football fields?
UPDATE: as sibling pointed out indirectly, it's eight thousand Popocatépetls [0].
[0]: https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28thic...
Placed in a grid, they'd cover an area larger than Wales [2].
Piled on top of each other, they'd reach a tenth the distance to the moon [3].
---
[1] https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28leng...
[2] https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28area...
[3] https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28thic...
That said, if I'm ever fuck-you rich, I'm going to have a pyramid built to bury me in and a library of hardcover printed wikipedia.
This is the individual's work files of civil servants. These will overwhelmingly be temporary documents they were legally obliged to delete at some point in the last 8 years. Any official filings or communications would have been to systems of record that were not effected.
This is more that a very large fire, probably unlucky for once a decade, caused civil servants to lose hours of work in files they were working on. A perfect system could have obviously prevented this and ensured availability, but not without cost.
No offsite backups is a real sin, sounds like a classic case where the money controllers thought 'cloud' automatically meant AWS level redundant cloud and instead they had a fancy centralized datacenter with insufficient backups.
Its almost weirdly a personality trait that a trained programmer just goes around believing everyone around him doesnt understand what way the wind blows.
Government installation for backups for a government ruled by a weird religious sect, have no offsite backups, it goes up in flames? Well clearly they were not smart enough to understand what an off-site backup is.
Its like wtf guys?
Now dont get me wrong, occams razor, they tried to save a few bucks, it all went Pete tong, but cmon, carelessness , chance, sure, but I doubt its all down to stupidity.
People, in general, severely overestimate their own intelligence and grossly underestimate the intelligence of others.
Consider for a moment that most of the geniuses on hacker news are not even smart enough to wonder whether or not something like IQ is actually a meaningful or appropriate way to measure intelligence, examine the history of this notion, question what precisely it is we mean by that term, how its use can vary with context, etc. etc.
Just wondering what it would be, just "success" in a domain?
I agree with you just wondering
The problem is more likely culture, hierarchy or corruption. Guaranteed several principal security architects have been raising the alarm on this internally along with much safer, redundant, secure alternatives that came with an increased cost. And decision makers who had a higher rank/social/networking advantage shot it down. Maybe the original storage designer was still entrenched there and sabotaging all other proposals out of pride. Or there's an unspoken business relationship with the another department providing resources for that data center that generates kickbacks.
Assuming nobody knows how to do an offsite backup or is plain ignorant of risk over there is arrogant.
ONE IS NONE
Sir Humphrey: Well, this is what we normally do in circumstances like these.
Jim Hacker: (reading) This file contains the complete set of papers, except for a number of secret documents, a few others which are part of still active files, some correspondence lost in the floods of 1967… Was 1967 a particularly bad winter?
Sir Humphrey: No, a marvelous winter. We lost no end of embarrassing files.
You tack on the hours required to do it yourself (this includes the time you must spend actually restoring from the backups to verify integrity, anything less can not be trusted). You keep one copy in your safe, and another copy in a safety deposit box at the bank. Nobody ever has to know. It is inevitable that you will save your own ass, and theirs too.
Shit happens.
this is the kind of thing that is so fundamental to IT that not doing it is at best negligence and at worst intentional malpractice. There is simply no situation that justifies not having backups and I think it might be worth assuming intentionality here, at least for purposes of investigation. It looks like an accident but someone (perhaps several someones, somefew if you will) made a series of shriekingly bad decisions in order to put themselves in a precarious place where an accident could have an effect like this.
New caption:
> A firefighter wants to see the cool explosive reaction between water and lithium at the National Information Resources Service (NIRS) in Daejeon on Sept. 27. [YONHAP]
https://www.datacenterdynamics.com/en/news/858tb-of-governme...
- G-drive stands for Government Drive
- The incident was caused due to Lithium battery fire
- The drive was of 858TB capacity
- No backup because “The G-Drive couldn’t have a backup system due to its large capacity” (!!)
I'm going to check all the smoke detectors in my house tomorrow :D
benoau•4mo ago
Yikes. You'd think they would at least have one redundant copy of it all.
> erasing work files saved individually by some 750,000 civil servants
> 30 gigabytes of storage per person
That's 22,500 terabytes, about 50 Backblaze storage pods.
Or even just mirrored locally.
yongjik•4mo ago
It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month).
They did have backup of the data ... in the same server room that burned down [2].
[1] https://www.hankyung.com/article/2025100115651
[2] https://www.hani.co.kr/arti/area/area_general/1221873.html
(both in Korean)
paleotrope•4mo ago
poly2it•4mo ago
lukan•4mo ago
BolexNOLA•4mo ago
sneak•4mo ago
BolexNOLA•4mo ago
Edit: my bad backups in the room is something, somehow just forgot about that part
sneak•4mo ago
SamPatt•4mo ago
username332211•4mo ago
Having had unfortunate encounters with government IT in other countries I can bet that the root cause wasn't the national culture. It was the internal culture of "I want to do the same exact same thing I've always done until the day I retire."
Absent outside pressure, civil services across the word tend advance scientifically - one funeral (or retirement) at a time.
rvba•4mo ago
Is their cost per unit so low?
Ekaros•4mo ago
0.00099*1000 is 0.99. So about 12$ a year. Now extrapolate something like 5 year period or 10 year period. And you get to 60 to 120$ for TB. Even at 3 to 5x redundancy those numbers start to add up.
vbezhenar•4mo ago
hapanin•4mo ago
vbezhenar•4mo ago
That's very primitive explanation, but should be easy to understand.
In reality S3 uses different algorithm (probably Reed-Solomon codes) and some undisclosed number of shards (probably different for different storage classes). Some say that they use 5 of 9 (so 5 data shards + 4 parity shards which makes for 80% overhead), but I don't think it's official information.
gundmc•4mo ago
alexjurkiewicz•4mo ago
[1] https://bigdatastream.substack.com/p/how-aws-s3-scales-with-...
burnt-resistor•4mo ago
sudo_and_pray•4mo ago
Now if you want to do something with the data, that's where you need to hold your wallet. Either you get their compute ($$$ for Amazon) or you send it to your data centre (egress means $$$ for Amazon).
npteljes•4mo ago
Imustaskforhelp•4mo ago
Yes its pricey but possible.
Now its literally impossible.
I think that AWS Glacier at that scale should be the thing preferred as they had their own in house data too but they still should've wanted an external backup and they are literally by the govt. so they of all people shouldn't worry about prices.
Have secure encrypted backups in aws and other possibilities too and try to create a system depending on how important the treat model is in the sense that absolutely filter out THE MOST important stuff out of those databases but that would require them to label it which I suppose would make them gather even more attention to somehow exfiltrate / send them to things like north korea/china so its definitely a mixed bag.
my question as I said multiple times, why didn't they build a backup in south korea only and used some other datacentre in south korea only as the backup to not have to worry about encryption thing but I don't really know and imo it would make more sense for them to actually have a backup in aws and not worry about encryption personally since I find the tangents of breaking encryption a bit unreasonable since if that's the case, then all bets are off and the servers would get hacked too and that was the point of phrack with the advanced persistent threat and so much more...
are we all forgetting that intel has a proprietory os minix running in the most privileged state which can even take java bytecode through net and execute it and its all proprietory. That is a bigger security threat model personally to me if they indeed are using that which I suppose they might be using.
npteljes•4mo ago
lucb1e•4mo ago
mastax•4mo ago
ycombinatrix•4mo ago
mastax•3mo ago
maxlin•4mo ago
_Only_ the kind of combination of incompetence and bad politics here can lead to the kind of % of how much data has been lost here, given the policy was to only save stuff on that "G-drive" and avoid local copies. The "G-drive" they intentionally did not back up because they couldn't figure out a solution to at least store a backup across the street ...
rasz•4mo ago
or outright buying hardware capable of storing 850TB for the same $20K one time payment. Gives you some perspective on how overpriced AWS is.
swarnie•4mo ago
I had 500TB of object storage priced last year and it came out closer to $300k
kachapopopow•4mo ago
HHad3•4mo ago
You of course need people to maintain it -- the $300k turnkey solution might be the better option depending on current staff.
mort96•4mo ago
jeroenhd•4mo ago
TheRoque•4mo ago
mrbadguy•4mo ago
axus•4mo ago
baobabKoodaa•4mo ago
jeroenhd•4mo ago