I'm happy for competition, but this seems a bit foul since we users aren't getting anything tangible beyond the promise of improvements and investments that I don't need.
Given that GitHub runners are still slow as ever, it actually is a point in our favor even compared to self-hosting on aws etc. However, it makes the value harder to communicate <shrug>.
I'm sure we'll feel it too at https://sprinters.sh, but probably a bit less than others as our flat $0.01 per job fee for runners on your own AWS account will still work out to about 80% average savings in practice, compared to ~90% now when using spot instances.
Of course, if you can just fence in your competition and charge admission, it'd be silly to invest time in building a superior product.
> Actions is down again, call Brent so he can fix it again...
Not sure if a Phoenix Project reference, but if it is, it's certainly in keeping with Github being as fragile as the company in the book!
The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes.
Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.
When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast.
Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.
Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.
Using GitHub actions to coordinate the Valhalla builds was a nice-to-have, but this is a deal-breaker for my pull request workflows.
A lot of it is wasted in build time though, due to a lack of appropriate caching facilities with GitHub actions.
[0] https://github.com/Barre/ZeroFS/tree/main/.github/workflows
tl;dr uses a local slot-based cache that is pre-warmed after every merge to main, taking Sidecar builds from ~10-15 minutes to <60 seconds.
SlateDB, the lower layer, already does DST as well as fault injection though.
I’m definitely sure it’s saving me more than $140 a month to have CI/CD running and I’m also sure I’d never break even on the opportunity cost of having someone write or set one up internally if someone else’s works - and this is the key - just as well.
But investment in CI/CD is investing in future velocity. The hours invested are paid for by hours saved. So if the outcome is brittle and requires oversight that savings drops or disappears.
(People seem to object to this comment. I genuinely do not understand why.)
Perhaps that isn't most use of it; the big projects are really big.
Fundamentally, yes, what you run in a CI pipeline can run locally.
That's doesn't mean it should.
Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally.
There are a rather lot of people who do argue that? Like, I actually agree that non-local CI is useful, but this is a poor argument for it.
I'm not aware of people arguing for self-hosting team or enterprise services.
Actions let you test things in multiple environments, to test them with credentials against resources devs don't have access to, to do additional things like deploys, managing version numbers, on and on
With CI, especially pull requests, you can leave longer running tests for github to take care of verifying. You can run periodic tests once a day like an hour long smoke test.
CI is guard rails against common failure modes which turn requiring everyone to follow an evolving script into something automatic nobody needs to think about much
... Is nobody in charge on the team?
Or is it not enough that devs adhere to a coding standard, work to APIs etc. but you expect them to follow a common process to get there (as opposed to what makes them individually most productive)?
> You can run periodic tests once a day like an hour long smoke test.
Which is great if you have multiple people expected to contribute on any given day. Quite a bit of development on GitHub, and in general, is not so... corporate.
I don't deny there are use cases for this sort of thing. But people on HN talking about "hosting things locally" seem to describe a culture utterly foreign to me. I don't, for example, use multiple computers throughout the house that I want to "sync". (I don't even use a smartphone.) I really feel strongly that most people in tech would be better served by questioning the existing complexity of their lives (and setups), than by questioning what they're missing out on.
The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves.
Workflows on the other hand, they have problems. The whole design is a bit silly
been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered
we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
In my experience gitlab always felt clunky and overly complicated on the back end, but for my needs local forgejo is better than the cloud options.
Part of this is fair since there is a cost to operating the control plane.
One way around this is to go back to using check runs. I imagine a third party could handle webhooks, parse the .github/workflows/example.yml, then execute the action via https://github.com/nektos/act (or similar), then post the result.
It’s been awhile since I looked. What’s a good alternative?
Jenkins has been rock solid, we are trying to migrate to Argo Workflows/Events, but there are a complaints (like deploying argo workflows with helm, such fun!)
https://tangled.org/tangled.org/core/blob/master/docs/spindl...
(no affiliation)
---
Blog post about Tangled's CI: https://blog.tangled.org/ci
https://bsky.app/profile/tangled.org
There looks to be a blog post here: https://blog.tangled.org/ci
I'm not a fan of nix and would have picked containers regardless for a git forge CI offering
I get that self-hosted runners generate huge egress traffic, but this is still wild. Hope it pushes more companies to look into self-hosted Gitea / Forgejo / etc.
Holy s***
That's more expensive than an m8i.large.
What on earth.
I realise 100% utilisation isn't realistic, but that still sounds very expensive when you're already BYOB.
It's worse than unrealistic. It's ludicrous. Any company running more than an hour of actions workflows per week on GitHub can afford a few dollars a month for infrastructure. The per-minute charge is less than the cost of a millisecond of engineering labor time.
Now the only alternative is to move builds, CI, etc. off of GitHub's platform entirely, and maybe your source control as well. In other words, a big pain. Github seems to have entered peak encrapification: the point where they openly acknowledge rent-seeking as their product approach, fully deprecating "building the best, most reliable, trustworthy product." Now it's just "Pay us high margins because the effort to migrate off is big and will take too long to break even."
Basically the modern day Heroku business model.
GitHub still supports e.g. PR checks that originate from other systems. We had PR checks before GHA and it's easy enough to go back to that. Jenkins has some stuff built in or you can make some simple API calls.
It's not as convenient, but it works just fine.
From this perspective this is a huge price jump, but self-hosting to save money can still work out.
Honestly, GitHub Actions have been too flaky for me and I'm begrudgingly reaching for Jenkins again for new projects.
[1] https://instances.vantage.sh/aws/ec2/m7i.large?currency=USD&...
> Hosted Agents > > 2,000 minutes/month
:-o
8h job is definitely more expensive to them than a 1 minute one, but I'd guess that the actual reason is that this way they earn more money, and dissuade users from using a third party service instead of their own runners.
the only rational outside rationale is this has the best financial projections, equitability with the customer be damned
gotta make up for those slumping ai sales somehow, amiright?
This isn't aimed at people actually self-hosting; it's aimed at alternative hosted runners providers. See this list
It seems GitLab has a much better experience in this department, but their pricing is hard to justify for us...
Genuinely curious if folks here had better experiences or recommendations for a smooth CI/CD experience.
The GitHub encrapification finally affects me. I am militantly unwilling to pay per minute to use my own computer. Time to leave. I can trigger and monitor builds myself thank you very much.
What an incredibly silly accusation to make of a company/service that streams movies and television. Like you understand it is possible to dilute the concept of civic responsibility right?
Companies don't care about society, unless it affects profit. Companies are not people, they are cold machines that through different means try to reach the same purpose, make more money.
No one should anthropomorphize companies. They might look like they have human qualities, same way like the T800 in the Terminator looked human.
It'll be interesting to see how this affects third party companies providing GitHub runners.
Now the playing field is more level, yay. Fun for those who choose to migrate away.
Charging per minute for self-hosted runners seems absolutely bananas!
to top it all off, they round up to the nearest whole minute instead of billing for actual usage which i assume they'll use for this new charge.
Earthly did not work out, and dagger had the problem of we support everything but but nothing is great
Today it's possible to spin up a company that sells GitHub Actions runners with a lower price and higher performance than GitHub's own hosted runners. These new fees will make that a lot less economically viable.
1. Services like WarpBuild (I'm the founder) are still cheaper than GitHub hosted runners, even after including the $0.002/min self-hosting tax.
2. The biggest lever for controlling costs now is reducing the number of minutes used in CI. Given how slow Github's runners are, or even the ones on AWS compared to our baremetal processor single core performance + nvme disks, it makes even more sense to use WarpBuild. This actually makes a better case for moving from slow AWS instances running with actions-runner-controller etc. to WarpBuild!
3. Messaging this to most users is harder since the first reaction is that Github options make more sense. After some rational thought, it is the opposite.
https://docs.github.com/en/actions/reference/workflows-and-a...
(Which, yes, has implications for energy use/climate change too for sure).
It doesn't look like i currently have access to the usage data on any of the lots-of-runners-lots-of-PRs projects I currently work on (which are still probably way less than some large companies).
Any "large companies" don't give a shit about things at this cost level. They spend more on the time it takes you to open the door. The number of CI minutes could be astronomical and it still wouldn't rate above the threshold of caring. The time people in this thread have spent wringing their hands is way more expensive.
On my larger organization, we have on average 20 to 30 *active* runners during business hours. Assuming 5 on the off-hours, my napkin math says it comes down to about 10 fully-utilized-runners per month, so about 864$/mo. For the size of my organization that is honestly totally acceptable.
This is assuming 0.002$ per minute of job being actively executed. If it turns out to be 0.002$ per minute of *runner being registered* on the control plane, it would increase quite a bit. We are still using the old HorizontalRunnerAutoscaler with actions-runner-controller, with quite a pool of prewarmed runners idling to pick up a job. It would be a strong reason to use the new RunnerScaleSet (to take advantage of the reactive webhook-based scaling) and keep a very lean pool of prewarmed runners.
This is not new, not unexpected. This is ongoing. Nothing stops this because who wins elections? How do they pay for all that publicity. Certainly "contributing" to campaigns is much cheaper than paying your taxes.
Supposedly this is a place for hackers. Hackers can build a better alternative.
I suspect we'll be doing that sometime in January or February.
I guess forgejo is the easiest migration path? https://forgejo.org/
1. Self-hosting runners is still cheaper than not Despite the $0.002/minute self-hosted runner tax, self-hosting runners on your cloud (aws/gcp/azure/...) remains the cheaper option.
2. Prefer larger runners If your workflow scales with the number of vCPUs, prefer larger runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, using actions-runner-controller with heavy jobs running on 1 vcpu runners is not a good idea. Instead, prefer a 2vcpu runner (say) if it runs the job ~2x faster.
3. Prefer faster runners All else being equal, prefer faster runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, if you're self-hosting on aws and using a t3g.medium runner, it's better to use a t4g.medium runner since the newer generation is faster, but not much more expensive.
4. Prefer fewer shards If you have a lot of shards for your jobs (example: tests on ~50 shards), consider reducing the number of shards and parallelizing the tests on fewer but larger runners.
5. Improve job performance This is not new advice, but it's now more important than ever because of the additional GitHub self-hosted runner tax.
6. Use GitHub hosted runners for very short jobs For linters and other very short jobs, it's better to use GitHub hosted runners.
Note: I make WarpBuild, where we provide github actions runner compute. Our compute is still cheaper than using github hosted runners (even with the $0.002/min tax) and our runners are optimized for high performance to minimize the number of mins consumed. I'm generally biased, but I think the points 1-6 apply irrespective of WarpBuild.
I hate GH Action runners with a passion. They are slow, overpriced, and clearly held together with duct tape and chewing gum. WarpBuild, on the other hand, was a breeze to setup and provided faster runners and lower prices.
This is a really shitty move.
Hey GitHub, your Microsoft is showing...
Given github ran 11.5 billion mins of actions in 2025, and most of them would've been on self-hosted runners, this move makes some sense from their POV.
However, this is still an... interesting... move, especially after bitbucket got all that hate a few weeks ago for doing something similar.
However, my experience with GitHub Actions was really poor. Some build that would run perfectly on my local machine and any other servers we have hosted would always time out on GitHub runners. I went back and forth from small runners to large runners and the result was always the same. Then I found that there are third-party companies just offering replacement runners for GitHub Actions at less than half the price for an amazing reliability and cost. It was a night and day difference.
Now... this move by GitHub is almost unbelievable. Charging folks to use their own machines
We currently self-host on kubernets/aws. The thing that really got to me isn't the new charge per se. It's the fact that GHA has a ton of problems. I can hold my nose and deal with them when it's free. But now that you're squeezing me, at least you could have created something like GHA 2.0 and added a charge for that. Instead, there are vague roadmap promises which don't even include things that I care about. Specifically:
- Jenkins had better kubernetes integration years ago. It's crazy that GHA can't beat that.
- "Reintroducing multi-label functionality" - yeah, so they first broke it. They did supply "reasons", which looked like they never talked to a customer. [1]
- Still no SDK of any kind.
- "Actions Data Stream" - or you can just fix your logging.
There are dozens more complains, which are easy enough to find. This kind of an approach just makes me want to make sure that I don't use GHA again. Even if I end up paying another vendor, at least I'll be treated as a customer.
[1] - https://github.com/orgs/community/discussions/160682#discuss...
"Thank you for your interest in this GitHub action, however, right now we are not taking contributions.
We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in."
Apples to oranges, naturally, but like this, our infra-jenkins master would pay for itself in hosting in a week of ansible integration testing compared to what GHA would cost. Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.
And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
I have cron jobs on several github projects that runs once a day and I have never been charged anything for it (other than my github membership). Should I expect to be charged for this?
Anyone using GitLab or any other VCM that you'd recommend? I'm absolutely done with Github. Or is everything else just as bad?
And the best (maybe?) part in your case is that the CI is based on GH Actions, so you can probably reuse your workflows without the need to adapt them.
For a company, I'd recommend self-hosting forgejo (which also has actions), which powers codeberg.
(forgejo started as a fork of gitea)
Alternatively, Forgejo, Gitea, or (based on praise I've seen from other people) maybe sourcehut.org.
I find GitLab's interface intolerable. Nonintuitive organization, overreliance on javascript, common operations hidden behind menus, mystifying icons used as placeholders... Every time I seek out a project's home and discover a GitLab instance, I find myself pausing to reconsider whether contributing to the project will really be rewarding enough to outweigh the unpleasant experience I'm about to have.
What does VCM stand for?
Focus on the enterprise. Something like a 3000$ minimum yearly price. Direct customer support with real engineers no questions asked.
Need someone to setup your CICD, that's another fee, but on staff engineers will get it done.
Edit: I'd even imagine a company like this can bootstrap, I'd need help though. Would probably take 4 skilled SWEs about 6 months for an MVP.
* Codeberg https://codeberg.org/
* Sourcehut https://sr.ht/ [1]
GitHub's log streaming also sucks. It's very laggy and chunked, whereas GitLab's is pretty much real-time.
I'm in the era of writing my own tools, not to share just for me or whatever group I'm working in. If you're going to charge me for something rife with annoying struggles, I might as well be annoyed by a tool I control.
To spell it out: jobs can hang forever because of some ridiculously bad code on their end, they have a 6 hour cap, so that's 6 hours of billable $$$ per-instance of the bug (assuming it wasn't manually canceled). I know I've seen jobs hang forever regularly over the course of my years using GitHub for work.
Note: pretty sure this has been resolved.
We will continue to do our best to provide the fastest GHA runners and keep them cheaper than GitHub-hosted runners.
To solve the problem, I created a simple vm in Google Cloud with a lot of CPU and memory that runs Ubuntu. I installed enough stuff on it to be able to check out code and run our build script (a jvm and gradle basically). And then I modified the Github action to 1) start the vm, 2) trigger the build script via ssh 3) pause the vm so we don't get billed for it. That vm runs for maybe an hour per month or so. It would probably cost us hundreds of euros per month if we ran it 24/7. But 1/3600th of that barely registers on our bills. And it's nice and fast.
This has been working flawlessly for a few years now. The Github action takes about 3 minutes. That includes starting the vm, running the script, and shutting the vm down again.
Wonky in a way. But also simple and robust enough. People over engineer/over think this stuff for the wrong reasons. For example, I could of course automate the provisioning of that vm. But I haven't. Because I only ever touch it once a year or so to run a quick apt-get update. I rebuilt it a few weeks ago in a different region. That was like a 20 minute job. Terraform or Ansible for vms you only create once every few years is redundant and might take more time than you would save. I can always do that when that stops being true.
I've been running this startup on the freemium layer in Github for five years now. It's great as a free service. I would actually pay for it if I needed to. I did actually pay for it before MS acquired Github in a previous startup when business usage wasn't free. But so far, there's no need for me to do that. I also run some monitoring scripts as Github actions. Simple curl jobs against our servers that trigger alerts when they fail. That has to run somewhere. It might as well be Github actions. But if/when that becomes inconvenient, I can improvise other solutions.
As far as I can tell from that article, these changes will not affect me; it says "Standard GitHub-hosted or self-hosted runner usage on public repositories will remain free" and another section says "This will not impact Actions usage in public repositories". Hopefully, this information would behelpful for other people who use GitHub Actions. However, I don't know if I missed something else that is important, from the article.
Arcuru•3h ago
Charging for self-hosted runners is an interesting choice. That's the same cost as their smallest hosted runners [1]
[1] - https://docs.github.com/en/billing/reference/actions-runner-...
tom1337•3h ago
Edit: Confused GitLab and Bitbucket
nstart•2h ago
I don't know if it's worth the amount they are targeting, but it's definitely not zero either.
acdha•2h ago
xp84•2h ago
franktankbank•1h ago
jeduardo•2h ago
tom1337•2h ago
jeduardo•2h ago
swatcoder•2h ago
ZIRP ended, its remaining monopoly money has been burnt through, and the projected economy is looking bleak. We're now in the phase where everything that can be monetized is being monetized in every way that can be managed.
Free tiers evaporate. Fees appear everywhere. Ads appear everywhere, even where it was implied they wouldn't. The lemons must be squeezed.
And because everybody of relevance is in that mode, there's little competitive pressure to provide a specific rationale for a specific scheme. For the next few years, that's all the justification that there needs to be.
wiether•1h ago
I thought that "Bitbucket" was in your original post and you added only your edit message to say that it was, in fact, Gitlab and not Bitbucket that added cost for self-hosted runners.
thewisenerd•2h ago
(ofc, that'd only mean they stop updating the status page, so eh)
teach•2h ago
https://downdetector.com/status/github/
falsedan•31m ago
IshKebab•2h ago
https://github.com/neysofu/awesome-github-actions-runners
sylens•1h ago
NewJazz•1h ago
Really Dianne?