Imagine: you start working at the company with a good will towards them. As you learn more and see more decisions, you slowly realize how bad it is. Now you've already contributed. Then you leave.
It might be possible to build everything with this goodwill and that's how we'd still end up with a torment nexus even if you immediately drop it.
What I'm saying is, at one point you have to break the company fully.
I'm not sure what more we need to wait for, if the claim "10% it ends humanity" is not enough.
Good people can be in tough position where even when they do not agree they may not have much of an option but to keep going.
What I wanted to say is that usually people start with good intentions and only later they realize they are contributing to a torment nexus.
What you are saying definitely plays a role and it makes doing moral choices difficult.
Yes, some members of the group don't have that freedom. But that doesn't excuse the ones who do.
> If the answer is the salary, say it louder, and do not force yourself to make excuses.
Its the same playbook, over and over again.
People do things.
Good or bad is relative to their own biases/experiences/ beliefs and moral compass.
Very few people are truly sociopaths who do evil stuff for personal gain.
Most do it because of the curse of the “banality of evil” (re: <https://en.wikipedia.org/wiki/Eichmann_in_Jerusalem?wprov=sf...>)
Doesn’t make it right, but like everything else in life the reality is much more complex than saying “good people refuse to do bad things”
My current read is that ~20-30% of people in America actively delight in hurting others (ICE supporters etc)
Another 30% do not seem to care about anything but their own well-being. (I don’t like to think I fall here but as a big tech peon it doesn’t look good).
The remaining percent seem to care and are trying their best but looking at how things are going it’s not enough
I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.
Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).
I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.
I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.
The deeper underlying thing seems to be some kind of mathematical mysticism.
Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.
It's not context-free limit-free magic. It's not special. It's an adaptation like any other.
& ofc AI is making climate change worse, some of the Amazon datacenters are using fossil fuels to generate a country’s worth of power, but that’s never part of their calculations
Good lord. You could apply the same logic to any taxpaying resident of any country whose government is doing something bad. There's some truth to it, it's worth bearing in mind, but it's also the very definition of an Isolated Demand for Rigor.
That is a fantastic term for something that seems to come up a lot, and I had not heard before.
The man is the definition of a useful idiot https://slatestarcodex.com/2013/08/30/military-strikes-are-a...
You can't, because those taxes are coerced from you at literal gunpoint if you refuse (or your property seized at gunpoint to the same effect). It's completely different than choosing to do that of your own free will.
When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for, but nobody is morally tainted for being forced to contribute a small slice to something evil. Maybe there's a threshold where you should be actively laying down your life to resist, eg. if being forced to commit actual murder, but I don't think you can reasonably expect that of paying taxes at all.
---
Edit: rate-limited, attaching a reply to a below comment.
> is that you also reduce the pool available for things you think are good
Indeed, but I also chose to contribute money to other good causes instead. In that way, the same sum could be used for 100% good instead of, say, 80% good 20% bad.
Now that I live in a country of my choosing, I am proud to pay taxes because of how they are used. 0% of my tax money goes to killing innocent people overseas, for instance.
The rule of all games played in capitalism is there's a sucker born every minute. you can work a few years or your whole life before you realize you've been had.
and most do....
It would be nice to see this survey repeated across professions. In my experience, most people have gripes about their profession because it’s what they know. Only half of respondents saying they would steer newcomers away feels a lot better than what my friends in medicine say all the time. Among my college professor and professor-track friends, this number probably reaches 100%.
Life isn't black and white with "good/bad". Further, there's no objective measure of what is good or bad. Everything is a shade of gray.
You certainly could argue that some actions are absolutely good or absolutely bad, but those aren't the majority of actions people take on a day to day basis. People aren't often faced with "should I kill this person" or "should I publish the cure for cancer". Instead it's "should I eat a hamburger or a salad" or "Should I finish this jira case or that jira case".
For example, you could argue that anyone working in an automotive plant is doing something bad because they are helping to produce vehicles which burn fuel which perpetuates climate change. But those plants also often produce parts for farm equipment or vehicles used to transport necessities like food.
People working at AI companies can make similar personal calculations. Like yeah there are bad things happening because of AI, but also they can believe that there are good things that can come as a result such as research and product developments.
Balancing good against bad is not clear, I agree, but that makes it all the more important to think about it when making a big decision such as what job to take.
Its hard and people do not even agree about how to balance it - e.g. whether the ends justify the means.
In the particular case of AI companies, I think a handbrake should be pulled when they start claiming things like "10% chance we're ending humanity."
So, there's an ojective measure of shade of gray?
He's still working in service of the AI companies though (same narrative) and no doubt will land a cushy job at so-callef "AI Safety" labs or make a living on the news/podcast circuit.
Here is one of his posts from 2022:
"Since a long time now I was interested in learning computer graphics and “do stuff” with computer graphics APIs. I began to work on a very simple Game engine in Swift / Metal last year but I began to switch to a Windows machine a few months ago, and Swift / Metal is not compatible with Windows (and will never be I think)."
https://carette.xyz/posts/opengl_and_cpp_on_m1_mac/
This is the same person who is apparently now writing:
"It is a transaction where you sell the part of you that was supposed to care. Do that often enough, and it does not come back. That is not burnout, but it is still a disease. Although, like every disease, it spreads: first to your team, then to your product, then to the people using it."
That said, I don't buy this article. I'm AI neutral. It's going to break a lot of things and fix many others. Good for you if you think you know what the net outcome will be but I think it's delusional to believe you do. Also I don't think you can resign and stop it without ceding the world to bad people. Pandora's box has already been opened. We've discovered a new thing math can do and now it's yet another arms race between various factions seeking to exploit it. Help the faction you want to win. It's not the world I want it to be, but it is what it is.
>So, why are you still working for them?
>If the answer is the salary, say it louder, and do not force yourself to make excuses.
>If the answer is “someone worse would replace me”, say it too, and louder too.
>If the answer is “I can change things from the inside”, then show me what you >changed. Haugen tried the inside. Coxon tried the inside. At the end, >unfortunately, the inside did not move.
Okay, but what if I think that both sides are LARPing their personal science fiction stories and reality will end up much more mundane (but still incredibly beneficial) than the utopias and dystopias everyone is cooking up?
This presupposing a moral dilemma that just doesn't exist in my opinion.
Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult.
If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world
I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults.
But what do I know. I'm probably low IQ.
2. Humans put it in charge of everything, as fast as possible, because it's just so damn useful
3. Is it perfectly aligned to human values for all of future ????
4. Everybody dies as a byproduct of the ASI pursuing some project that humans don't even have a chance to understand, like an ant colony during construction of a hydroelectric dam.
You should beware that you've rationalized everything you do as good enough, and everything that other people do is morally questionable.
Have you set the bar for what is 'reasonable' to be extremely convenient for your own circumstances?
Social security? Medicare?
The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good. Every country will do a mix of things you judge good, and things you judge bad.
> nobody is morally tainted for being forced to contribute a small slice to something evil
I agree.
I like how this is being spun as virtuous. When multinational trillion-dollar companies do it, we say it's bad!
But there are several workable contextual definitions of bad; it’s your job to know what they are.
For example:
- it would cause a harm to someone who was not made aware of the risk up front.
- it would cause someone to lose a job they didn’t know was under threat.
- the way the harm occurs precludes its only corrective/curative action or has no adequate compensation.
Bad things are almost always contextual but also almost always obvious in that context.
You can’t avoid bad things that you can’t foresee. If you do not attempt to avoid, avert or mitigate bad things that you can foresee, then you are one of the bad things.
Things may not always be black and white but it is wise not to seek the protection of the grey.
Under this guidance, wouldn't almost every technological advancement then qualify as bad? i.e. typewriter repair, etc.
jmclnx•1h ago
Story time :)
I remember when I was first starting out as a programmer. At QE (always occurred on a weekend) a slew of financial reports were created and printed for review the following Monday then I think used for summaries for Wall Street to look at.
A high level finance manager came to me and asked if the report could be changed, not knowing what he was asking I said sure. After a long discussion turned out he wanted the numbers on the report changed and re-printed. I said it was possible but the numbers would be all wrong.
So he said "do it". I went to my manager and told them what he wanted, then I refused to do it. I said I would quit first because I did not want the SEC to come after me. Luckily they all supported me and the finance manager was told to "FO".
FWIW, that company is long gone.
dofm•27m ago
I had to make this conditional promise once, back in the nineties, as a junior person in a related industry. Because in the dot-com era developers like me were often simultaneously a babe-in-arms on a professional career level and also the only person who knew how to do the work that was being sold. I decided that if it came to it, I would not be the person having a hard time in court. The client understood better than my bosses and they changed the product. Took them a few minutes at most, but I have reason to believe that my decision made some ripples.
Nowadays I don’t let those issues surprise me: I believe in establishing specific ethical standards at the beginning of projects, because if you don’t do that, you don’t have a basis to robustly point to actions that put them at risk.