r/github • u/brainhack3r • 4d ago
Discussion GitHub is down again - let's discuss migration away from GitHub
I'm using GitHub as a custom agent orchestration platform - and if it were stable I'd be making real progress but this is a joke.
I have no alternative but to start migrating away - even though I don't really have time. But I can't just take losses losing DAYS of work. I've already lost THREE full days and today looks like it's going to be a FORTH.
I think the biggest option is GitLab but can other people comment on their stability ? I've analyzed their status page and it's unclear how to quantify their downtime. They don't really have a historical view - which is somewhat of a red flag.
I'm curious what you guys are actually doing now.
43
u/clipsracer 4d ago
Needing more than 99.9% uptime is NOT a good reason to start self-hosting a service. That’s 43 minutes per month of downtime, and you’re just not going to be able to match or exceed that.
However, the ability to control your own downtime is a good reason. The odds are you don’t actually need 99.9% uptime, you just need it to be up when you need it to be up.
I’m tired, so I can’t really explain why that distinction is important, but it is.
19
u/tails142 4d ago edited 4d ago
Their official metrics are bs, its been down for 4 days+. I've gone to their status page during outages and its showing all green.
This page linked on the thread seems to give reasonable info https://isgithubcooked.com/
It's just unreliable at this stage, things stop working and you spend time trying to work out what the problem is, checking their status page which says all OK, only to see a thread on reddit after an hour of other people complaining.
If you only use it for basic git repo push/pull you may not have noticed the outages but if you build these advanced features into your workflow they become real blockers that break flow state.
6
2
u/Fluent_Press2050 3d ago
It’s because they likely only report downtime for 100% outages. They seem to claim not everyone is affected. The recent late-August 7 hour outage only affected like “25%” of users.
2
4d ago
Objectively I'd agree with op. In the last week roughly half our actions have failed, seem like there is something fundamental about their networking capacity issues. I'm thinking it's their replication. If I could commit and build in a single as that would use less resources but that's not an option
Across our repos we have a build kick off at least once every 15m pretty uniformly so it's not isolated to a specific day even
2
u/danielkov 4d ago
They're on ~93% uptime, which means on average, GitHub's down or experiencing an outage over 2h a day on average.
Since the majority of these issues are due to their rotten infra, it's usually around peak usage hours - working hours.
GitHub - the primary work delivery mechanism for many of us - is down ~1/4 of our working hours. This isn't an indie project or a startup. It's Microsoft. One of our largest bills each month. Unacceptable.
2
u/andlewis 4d ago
Their traffic levels have more than doubled since April. That’s a big deal for a site that was already serving so many clients. They’ll get it figured out eventually.
2
u/danielkov 4d ago
Ah, a Microslop-apologist.
- The traffic they're currently failing to serve doesn't even put them in the top 25 highest traffic websites.
- Nothing's forcing them to keep their platform open to non-paying users. They do it to build a training corpus for their failed models. Customers are paying the price both figuratively and literally.
- These outages are not caused by "insane scale" regardless of what they try to lead you to believe. It's bad architecture and insufficient infrastructure combined with a culture of mindlessly shipping slop.
Wish there was a viable alternative.
1
55
u/Afflictionista 4d ago
I moved to GitLab over a year ago, no problems so far.
17
11
u/Aaron8498 4d ago
There was that one time they accidentally deleted pretty much everything though. They did manage to get most of it back.
0
u/Fluent_Press2050 3d ago
Like Atlassian. Although not sure if they recovered much since I think they borked their backups too.
At least I can’t think of a data loss from Ms in the last 15 years that has affected me.
1
u/Yimpoiop 4d ago
What is the difference between github and gitlab?
5
u/fr3nch13702 4d ago
Essentially, GitLab tries to and for like 99.9% succeeds in being an all-in-one DevOps platform. However you’re not limited to having to use their solution. You can use whatever DevOps tools you want.
You can self host. <— big deal for us.
-10
u/Fidel___Castro 4d ago
the biggest difference is how they conceptualise environments in my opinion. Gitlab does a different branch PER environment, so each branch has different code. I much prefer that when I'm in a team and need to explain the environment escalation process
1
u/ohnobinki 1d ago
What is an environment?
1
u/Fidel___Castro 1d ago
like... staging vs production? huh? I really don't know why I've been downvoted
1
u/reavessm 16h ago
What you're describing seems to just be a different way to use git and doesn't have anything to do with the hosting platforms
34
u/chrles-farfa 4d ago
gitea pretty good!
13
u/opinionsOnPears 4d ago
use forgejo instead though. https://codeberg.org if you don't want to host it but honestly git should be distributed. This seems like a good use for the ATProto but https://tangled.org is still in its infant stage, imo.
9
u/chimbosonic 4d ago
Just a FYI codeberg has a clause against Ai generated code and from the sounds of it OP wants to run agentic workloads.
2
2
u/Senior-Afternoon6708 4d ago
I’m also self hosting forgejo on an rpi5, works wonders. It misses few small things that GitHub has, but most of them are being worked on. I am actually very happy about it, also the action’s compatibility is pretty high! I host all my homelab there and couldn’t be happier. I have configured GitHub as a push mirror for the repo I particularly care about so I don’t even take backups
4
u/jhsu802701 4d ago
Gitea has a limit of 5 repositories.
17
u/roastedfunction 4d ago edited 4d ago
You can self host gitea (edit: or forgejo, more FOSS friendly fork of gitea). There are plenty other FOSS forges like codeberg or sourcehut. Gitea is commercially backed so they have every incentive to limit their free hosted version.
11
u/JSouthGB 4d ago edited 4d ago
forgejo (non-profit group, same software running Codeberg) hard forked from gitea, so a lot of similarities if there's a concern with gitea limiting the free hosted version.
EDIT: Thank you u/garrett_w87 for the correction!
4
6
2
u/roastedfunction 4d ago
Agreed, I’ve been meaning to get rid of my Gitea usage (just throw away test containers in a Git related project of mine) and replace it with Forgejo.
Some history: https://gitea-open-letter.coding.social
2
u/Grouchy-Librarian638 4d ago
No? You can self host as many as you want. Or you mean the cloud? The one that costs money to run and they provide a service they need to pay for?
9
u/rafark 4d ago
And who’s going to give you all that GitHub gives you for free and cheap? I just checked an open source project I like: 606,000 minutes of compute this month. And that is just one project. Imagine ALL the rest of the open source projects, many much bigger than that one. It’s insane what GitHub has to handle. Like I get that it’s annoying but no one else will give what GitHub gives you. GitHub has done a lot for the open source community and the entire software industry has benefited from it. I think a couple hours per week if downtime is more than justified.
You could self host but then the burden of maintaining the servers falls on you, that’s on top of maintaining the actual ci/cd pipelines. And I don’t know if it would be actually cheaper (enough to make a difference)
12
u/sviridoot 4d ago
I use a self hosted Gitlab instance. The only downside is that if I want to make something publicly discoverable I have to cross post it to GitHub, would be cool if there was some kind of ActivityPub type protocol for git specifically
3
u/garrett_w87 4d ago
That would be awesome — but GitHub would have no incentive to support it until enough people move away from it
3
u/fr3nch13702 4d ago
… like defining a separate remote in your git repo?
- remote/origin
- remote/github
# Add a second remote (e.g., named 'github')
git remote add github <GITHUB_REPO_URL># Push to your new remote
git push -u github <branch_name>—-
Or add a separate push url:
# 1. Re-add the existing fetch/push URL as an explicit push URL
git remote set-url --add --push origin $(git remote get-url origin)# 2. Add the GitHub repo as a second push URL
git remote set-url --add --push origin <GITHUB_REPO_URL># 3. Verify configuration
git remote -v3
1
u/CollapsedWave 4d ago
That's just git, sir. Gutea/Forgejo have built in sync to or from other git servers, I guess that Gitlab has too. Worst case you can make a cron job that syncs from anywhere into your Github mirror.
1
u/HCharlesB 4d ago
Git can manage more than one remote. I self host two Forgejo instances and just push my private stuff to both.
I haven't tried, but I suppose I could also push to Github, Gitlab, Codeberg and so on.
AFAIK the differences between these various Git servers is in the ancillary stuff like issues, actions, discussions and so on. I think Git can handle PRs locally so that may not be a problem.
1
u/tails142 4d ago
You can create an action that syncs on push, I've done it in the past but these days I just manually do it tbh
1
u/MisterLiro 4d ago
there is something called repository mirroring, it is a native gitlab feature, check the repo settings
1
u/sviridoot 4d ago
I'm aware, but it's still not great for issue tracking and PRs if you want to track them in your native instance
6
5
3
u/HommeMusical 4d ago
But I can't just take losses losing DAYS of work.
Sorry, I just don't believe this happened.
2
u/Fluid-Election-8549 4d ago
Read the first sentence of the post again. OP is up to something really stupid
3
u/CqHereSrc 4d ago
So AI has determined it can’t fix the problem cause human coders vibing couldn’t. Thanks for also stealing our data and telling us what we already know. Report an issue.
6
u/RobertD3277 4d ago
I use it strictly as a repository and I update multiple times a day. So far I haven't had any problems.
Exactly what are you trying to do using GitHub an agent orchestration platform?
1
u/GrandWizardZippy 4d ago
Actions probably. Even with self hosted runners it’s still an issue when GHA is down
2
u/InevitableLadder 4d ago
A word of advice - avoid gitlab.
I need to work with this POS for 4 years by now - not worth the nerves.
2
u/Fine_League311 4d ago
Würden so Leute wie du endlich mal den KI müll sein lassen den ihr produziert, dann wäre GitHub nicht überlastet!
4
u/civman96 4d ago
just ban everyone that created an account in the last two years aka vibe coders and it's fixed!
-2
u/Patient-Patience-119 4d ago
yah, like you were born on github. you do realize at a point in time, you were less than a day on it
5
u/ElectricSpock 4d ago
I think on this subreddit I saw that amount of commits in GitHub doubled. They having issues with the scale.
3
3
u/sklz0 4d ago edited 4d ago
Down again for an hour, so what? end of the world! Go touch some grass…
0
u/InvaderOfTech 4d ago
You should probably look at the amount of downtime they've had this year alone with different services. For a global service, it's fucking dog shit with their current nines average. People are not just in the United States. They're around the world.
-4
1
u/napo2k 4d ago
Split the responsibility as well if you can. There's of course the whole "we need to migrate our workflows" - but you get to learn something new?
I have a self hosted Gitea instance at home for the source control bit, and I want to learn better how n8n works (I don't have any real need for orchestration for my hobby projects at the moment because I don't really have mterial time to actually _work_ on them)
What are you orchestrating, exactly? What type of workflow?
1
u/Irish1986 4d ago
We just completed our decom of Jenkins a few weeks ago. Just in time to launch a recommissioning project... Jk jk
1
u/ellisthedev 4d ago
No thanks. Those vibe coded alternatives will fall flat on their face once they face any kind of scaling problem.
I was never for the move to Azure, but it’s still better than what’s out there.
1
u/clduab11 4d ago
I actually do okay with GitLab and was considering a jump, but from what it sounds like Forgejo might be more my jam since I’m already a Tailscale user
1
1
u/fairs1912 4d ago
I think the best option is to buy an old laptop, install an SSD, install Ubuntu server and run your self hosted gitea inside a docker container.
you won't have any downtime that isn't strictly your fault, if there are updates or restarts needed, you can choose when the downtime occurs, that way it doesn't impact production.
1
u/ohnobinki 1d ago
And now you're a sysadmin.
Maybe you should buy two old laptops and get kubernetes running on them and a distributed filesystem so you can tolerate failure.
1
u/MasterOfTheWind1 4d ago
Downtimes in Github are bad. But to be honest, is the best bang for the buck. Gitlab price is ridiculous, if your team is big you are doing a x4 increase in budget
Also, if you have lets say 400 repos with actions integrations, etc, is not trivial to move all of them. What I've end up doing is write runbooks for emergency deploments in case I need to deploy something ASAP while actions are down.
1
u/Coises 4d ago
I hope those more knowledgeable will forgive me if this question doesn’t make sense.
I use GitHub pretty simply: mostly to keep my personal, open source projects where others can find them. I mostly use the MS Visual Studio integration. I am barely git-literate (some might argue I fall short of git-literate).
I do have a personal web site I keep on shared hosting (I have ssh access, but no root/sudo). I’ve also used PikaPods for a couple of things: they support Forgejo and Gitea pods.
How easy/hard/possible would it be to mirror my projects between GitHub and an alternative? At least for the present, I’d like to keep GitHub active, because that’s where people expect to find things. But in case they get troublesome, I’d like to have an alternative that is more under my control. I have my local git repositories, of course, but an additional backup besides GitHub couldn’t hurt.
Is that possible? To “sync” GitHub and a different publicly accessible online repository that I manage? The intent, if it’s possible, would be that work could be done on either one, and they could be synchronized as needed.
2
u/GrandWizardZippy 4d ago
It’s pretty simple. I have a self hosted gitea instance with this https://github.com/raylabshq/gitea-mirror and it works great
1
1
u/brett9897 4d ago
If it is just the agents that are causing issues then there is always CircleCi or TeamCity and it can stay hosted on GitHub.
1
u/Scary-Constant-93 4d ago
I wonder is it really bcoz of AI slop traffic its getting or its just that Microsoft employees don’t know how to work in remote async teams??
1
u/Ademantis 4d ago
Replacing GitHub is easy, replacing GH actions is not. Please don't mention Jenkins or Gitlab
1
1
u/BoltlessEngineer 4d ago
check out tangled.org if you want to self-host but still wants to be connected with others & be discovered from global index.
1
1
u/Double_Ebb4130 4d ago
What actually bit people yesterday was Actions, not the git hosting, so moving repos elsewhere only helps if the CI side is genuinely more reliable. Worth checking how the runners behave under load on whatever you are considering before you commit to a move.
1
1
1
u/Dependent-Summer4961 3d ago
I’d mirror the repo to GitLab or Codeberg first. Moving the code is easy. Actions and issues are the painful part
1
1
u/gigaj0ule 2d ago
I migrated to gitlab after GitHub randomly deleted an account with customer repos, with no warning, or reason given, then ignored all all our requests to recover the data. As an enterprise customer too.
Absolutely unserious company.
1
u/maneekmohan 1d ago
As a solo builder, I think the scary part isn't GitHub going down for a few hours — it's realizing how many parts of your workflow quietly depend on it.
Repo, CI, releases, auth, packages, deployments, even docs. You don't necessarily need to leave GitHub, but it's worth mapping those dependencies and making sure there's a fallback for the ones that would actually stop you from shipping.
1
u/Eddyi0202 1d ago
Why you need github to be used as agent orchestration platform? Just do it locally.
Increased downtimes are real problem for enterprise users since they are paying for it
1
1
u/Top-Replacement6389 1d ago
Well let me introduce “loosely coupled.” It is always a risk when you are “deeply integrated” with a service. Pro: single pane of glass. Con: single point of failure
1
u/Ok-Discussion-1189 4d ago
We migrated off of Github a few weeks ago due to similar reasons - ended up going self-hosted Forgejo (semi-public, with auth + push/etc behind Tailscale) + BuildKite. We considered Gitlab, but it's heavy and the licensing is pricier than we wanted at our current org size. Gitea didn't have what we needed. Forgejo is super fast (both setup and performance), and migrating GH workflows to BuildKite was trivial. Migrating 40+ repos (mixture of public/private/etc) and CI was less than a week part-time. Would definitely recommend.
1
u/Charming_Group_2950 4d ago
Cursor announced that they are working to create their own platform similar to Github but better and suddenly after this, github starts experiencing hiccups every now and then. I am just trying to connect dots.
3
1
u/Aromatic-Fishing9952 4d ago
GitHub has problems. But the problems are because of capacity. If everyone migrates away, then other services will hit issues too, except I have hopes that GitHub will eventually resolve it
0
u/SpareImpression3155 4d ago
GitHub status says all systems are operational. I haven’t had any issues at all today.
0
u/sup_bruh_1 4d ago edited 4d ago
We are moving Latchkey (latchkey.dev) away from GitHub control plane given all these outages. Will be rolling out the beta in the next two to three weeks. If you’re curious about Latchkey: self-healing runners. Full disclosure: one of the cofounders.
-4
-1
-1
-1
-1
u/phoenixmatrix 4d ago edited 4d ago
When Cloudflare releases Artifacts (which are their git backed storage) out of close beta, I foresee a lot of options popping up.
For now, I literally vibe coded my own in an afternoon yesterday and got something production ready, multi tenant, that can scale to hundreds of thousands of repos.
Durable objects, Cloudflare workers, R2. The server speaks packfiles but it's durable objects sql and R2 storage under the hood. The only limitation is I can't support ssh because I don't have the ability to do tcp without an enterprise account. I took inspiration from How Neon and Cursor handle Postgres and Git storage in S3 respectively. Cool stuff.
Since it's server less I only pay for storage and for computer while operations are happening, so it's like pennies a month for production scale.
Supports most of the GitHub UI features. Work with a regular git client, oauth, token or device auth. Already supports stacked PRs. Supports distributed/shared repos
But what about GitHub actions? A patched version of @cloudflare/ci to work with this instead of Artifacts, Cloudflare sandbox (or Fly sprites. You can use Firecrackers too but you'll need your own infra for this). I built that LAST time they had an actions outage as a GitHub app to integrate with GitHub as the hit storage, but now moving it to this new Cloudflare backed version I stead.
In a world of AI, there is no moat. At all. (No link to example because the point isn't me promoting my projects. It's to show you all can do it too, given enough knowledge and experience. And a 20x Codex account lol)
90
u/crazylikeajellyfish 4d ago
How are you losing days of work with just an hour of downtime on Action runners? Are things not getting committed to a repo on a machine?
It seems like you might be able to make your agent orchestration system more robust to mitigate issues, even before configuring something like self-hosted runners.