If you'd have asked me a few years ago if anything could be an existential threat to github's dominance in the tech community I'd have quickly said no.
If they don't get their ops house in order, this will go down as an all-time own goal in our industry.
Of course they're down while I'm trying to address a "High severity" security bug in Caddy but all I'm getting is a unicorn when loading the report.
(Actually there's 3 I'm currently working, but 2 are patched already, still closing the feedback loop though.)
I have a 2-hour window right now that is toddler free. I'm worried that the outage will delay the feedback loop with the reporter(s) into tomorrow and ultimately delay the patches.
I can't complain though -- GitHub sustains most of my livelihood so I can provide for my family through its Sponsors program, and I'm not a paying customer. (And yet, paying would not prevent the outage.) Overall I'm very grateful for GitHub.
Which security bug(s) are you referring to?
have you considered moving or having at least an alternative? asking as someone using caddy for personal hosting who likes to have their website secure. :)
We can of course host our code elsewhere, the problem is the community is kind of locked-in. It would be very "expensive" to move, and would have to be very worthwhile. So far the math doesn't support that kind of change.
Usually an outage is not a big deal, I can still work locally. Today I just happen to be in a very GH-centric workflow with the security reports and such.
I'm curious how other maintainers maintain productivity during GH outages.
Yep, I get you about the community.
As an alternative, I thought mainly as a secondary repo and ci in case that Github stops being reliable, not only as the current instability, but as an overall provider. I'm from the EU and recently catch myself evaluating every US company I interact with and I'm starting to realize that mine might not be the only risk vector to consider. Wondering how other people think about it.
> have you considered moving or having at least an alternative
Not who you're responding to, but my 2 cents: for a popular open-source project reliant on community contributions there is really no alternative. It's similar to social media - we all know it's trash and noxious, but if you're any kind of public figure you have to be there.
Several quite big projects have moved to Codeberg. I have no idea how it has worked out for them.
N.I.N.A. (Nighttime Imaging 'N' Astronomy) is on bitbucket and it seems to be doing really well.
Edit: Nevermind, looks like they migrated to github since the last time I contributed
I get that, but if we all rely on the defaults, there couldn't be any alternatives.
You are talking to the maintainer of caddy :)
Edit- oh you probably meant an alternative to GitHub perhaps..
no worries, misunderstandings happen.
GitHub has had customer visible incidents large enough to warrant status page updates almost every day this year (https://www.githubstatus.com/history).
This should not be normal for any service, even at GitHub's size. There's a joke that your workday usually stops around 4pm, because that's when GitHub Actions goes down every day.
I wish someone inside the house cared to comment why the services barely stay up and what kinds of actions are they planning to do to fix this issue that's been going on years, but has definitely accelerated in the past year or so.
It's 100% because the number of operations happening on Github has likely 100x'd since the introduction of coding agents. They built Github for one kind of scale, and the problem is that they've all of a sudden found themselves with a new kind of scale.
That doesn't normally happen to platforms of this size.
A major platform lift and shift does not help. They are always incredibly difficult.
There are probably tons of baked in URLs or platform assumptions that are very easy to break during their core migration to Azure.
Looks like AI replacement of engineering force in action.
You're absolutely right! Sorry I deleted your database.
I can help you restore from backups if you will tell me where you backed it up.
You did back it up, right? Right before you ran me with `--allow-dangerously-skip-permissions` and gave me full access to your databases and S3 buckets?
You're right! Let's just quickly promote your only read replica to the new primar---oops!
Github is moving to Microsoft Azure which is causing all of this downtime AFAIK
That's cover. They've been doing that since microsoft bought them
Yeah but that's exactly the issue - that whole time dev time will have been getting chewed up on the migration when it could have been spent elsewhere
More like Tay.ai and Zoe.ai AIs still arguing amongst themselves not being able to keep the service online for Microsoft after they replaced their human counterparts.
Screw GitHub, seriously. This unreliability is not acceptable. If I’m in a position where I can influence what code forge we use in future I will do everything in my power to steer away from GitHub.
Forge feature parity is easy to find. But GH has that discover ability feature and the social queues from stars/forks.
One solution I see is (eg) internal forge (Gitlab/gitea/etc) and then mirrored to GH for those secondary features.
Which is funny. If GH was better we'd just buy their better plan. But as it stands we buy from elsewhere and just use GH free plans.
Stars are just noise. All they tell you is how online the demographics of that ecosystem are.
Mirroring is probably the way forward.
They're overwhelmed with all the vibecoded apps people are pushing after watching the Super Bowl.
Their network stack is ran by OpenAI and is now advertising cool new ways for us to stay connected in a fun way with Mobile Co (TM).
Sorry, my fault. I tried to download a couple of CppCon presentations from their stash. Should have known better than to touch anything C++. ducks
pretty clear that companies like microsoft are actually terrible at engineering, their core products were built 30 years ago. any changes now are generally extremely incremental and quickly rolled back with issue. trying to innovate at github shows just how bad they are.
It's not just MSFT, it's all of big tech. They basically run as a cartel, destroy competition through illegal means, engage in regulatory capture, and ensure their fiefdoms reign supreme.
All the more reason why they should be sliced and diced into oblivion.
yeah i have worked at a few FAANG, honestly stunning how entrenched and bad some of the products are. internally, they are completely incapable of making any meaningful product changes, the whole thing will break
to be fair, git is one of the most easily replaced pieces of tech.
just add a new git remote and push. less so for issues and and pulls, but at least your dev team/ci doesn't end up blocked.
It's a general curse of anything that becomes successful at a BigCorp.
The engineers who build the early versions were folks at the top of their field, and compensated accordingly. Those folks have long since moved on, and the whole thing is maintained by a mix of newcomers and whichever old hands didn't manage to promote out, while the PMs shuffle the UX to justify everyones salary...
im not even sure id say they were "top", id more just say its a different type of engineer, that either doesnt get promoted to a big impact role at a place like microsoft, or leaves on their own.
Seems like MS copilot is vibe-ing it again ! Some other major cloud provider outages come to mind that never happened before the "vibe" area.
This one has more votes and comments though
On the plus side, it's git, so developers can at least get back to work without too much hassle as long as they don't need the CI/CD side of things immediately.
The biggest thing tying my team to GitHub right now is that we use Graphite to manage stacked diffs, and as far as I can tell, Graphite doesn't support anything but GitHub. What other tools are people using for stacked-diff workflows (especially code review)?
Gerrit is the other option I'm aware of but it seems like it might require significant work to administer.
I use git town. Fits my brain a lot better.
Is it really that much better than alternatives to justify these constant outages?
We're starting to have that convo in our org. This is just getting worse and worse for Github.
Hosting .git is not that complicated of a problem in isolation.
No, but it has momentum left over from when it was much better. The Microsoft downslide will continue untill there's no one left
Not any longer. It used to but the outages have become very common. I am thinking about moving all my personal stuff to Codeberg.
I love its UI (apart from its slowness, of course). I find it much cleaner than Gitlab's.
You can self-host GitHub enterprise.
Ooh - got a source?
It always has been to just self host. Predicted GitHub's outage streak as far back as half a decade ago [0].
"A better way is to self host". [0]
Im using Bitbucket for years with no issues.
The great advantage of Bitbucket is that it's so painfully slow you can't tell if it's down or not.
self-host your own services. There are a lot of alternatives to GitHub.
Oh! It's not my GitLab@Hetzner that's not working, it's GitHub. Just when I decided to opensource my project.
Well done for self-hosting.
I wonder what's the value of having a dedicated X (formerly Twitter) status account post 2023 when people without account will see a mix of entries from 2018, 2024, and 2020 in no particular order upon opening it. Is it just there so everyone can quickly share their post announcing they're back?
Looks like they've got a status page up now for PRs, separate from the earlier notifications one: https://www.githubstatus.com/incidents/smf24rvl67v9
Edit: Now acknowledging issues across GitHub as a whole, not just PRs.
It feels like GitHub's shift to these "AI writes code for you while you sleep!" features will appeal to a less technical crowd who lack awareness of the overall source code hosting and CI ecosystem and, combined with their operational incompetence of late (calling it how I see it), will see their dominance as the default source code solution for folks using it to maintain production software projects fade away.
Hopefully the hobbyists are willing to shell out for tokens as much as they expect.
It's a funny coincidence - I pushed a commit adding a link to an image in the README.md, opened the repo page, clicked on the said image, and got the unicorn page. The site did not load anymore after that.
List of company-friendly managed-host alternatives? SSO, auditing, user management, billing controls, etc?
I would love to pay Codeberg for managed hosting + support. GitLab is an ugly overcomplicated behemoth... Gitea offers "enterprise" plans but do they have all the needed corporate features? Bitbucket is a joke, never going back to that.
to be fair, i think usage has increased a lot because of coding agents and some things that worked well for now can't scale to the next 10x level.
Maybe they need to sort things out for people who pay through the nose for it cause I ain't comforted by vibe coders slowing us down.
Well its a day that ends in Y.
Github is down so often now, especially actions, I am not sure how so many companies are still relying on them.
Migration costs are a thing
So are the costs of downtime.
Just remove all that copilot nonsense and focus on uptime... I would like to push some code.
The saddest part to me is that their status update page and twitter are both out of date. I get a full 500 on github.com and yet all I see on their status page is an "incident with pull requests" and "copilot policy propagation delays."
They put too much AI in it bot enough engineering rigor
I don't know if it's related, but for the past week I've been getting pages cut off at some point, as if something closed the connection mid-transfer.
Today, when I was trying to see the contribution timeline of one project, it didn't render.
Anyone have alternatives to recommend? We will be switching after this. Already moved to self-hosted action runners and we are early-stage so switching cost is fairly low.
Codeberg, if your product/project is open source, otherwise try out Tangled.org and Radicle!!
Radicle is the most exciting out of these, imo!
So, what're people's alt stack for replacing GitHub?
We're mirroring to Gitea + Jenkins.
It's definitely some extra devops time, but claude code makes it easy to get over the config hurdles.
Codeberg, Tangled, Radicle!
Wait a minute, isn't Git supposed to be... distributed?
Yeah, but things with "Hub" in their name don't tend to be very distributed
Thanks for underscoring the beautiful oxymoron.
Issues, CI, and downloads for built binaries aren't part of vanilla Git. CI in particular can be hard if you make a multi-platform project and don't want to have to buy a new mac every few years.
Probably Worth taking an honest look at whether your CI could just be an SQS queue and a Mac mini running under your desk though
the incident has now expanded to include webhooks, git operations, actions, general page load + API requests, issues, and pull requests. they're effectively down hard.
hopefully its down all day. we need more incidents like this to happen for people to get a glimpse of the future.
And hey, its about the best excuse for not getting work done I can think of
We replaced everything except the git part because of reliability issues. Pages…gone Actions…gone KB…gone. Tickets…gone.
Maybe they need to get more humans involved because GitHub is down at least once a week for a while now.
Damn, I was also trying to push and deploy a critical bug fix that was needed within minutes.
Well unfortunately, you have to wait for GitHub to get back online to push that critical bug fix. If that were me, I would find that unacceptable.
Self hosting would be a better alternative, as I said 5 years ago. [0]
sorry all, i took a month off and then opened github.com
Microslop strikes again!
1 engineer, 1 month, 1 million lines of code.
Has anyone noticed that in the past year we have seen a LOT of outages?
Yes. Feels like every other week.
That goes against all the gushing posts about how AI is great. I use all the frontier models and sure they're a bit helpful
But I don't understand if they're that good why are we getting an outage every other week? AWS had an outage unsolved for about 9+ hrs!
Do they publish proper post-mortems? I feel like that's gotta be the bare minimum nowadays for such critical digital infrastructure.
The new-fangled copilot/agentic stuff I do read about on HN is meaningless to me if the core competency is lost here.
I look forward to the day that jjhub becomes available...
Azure Screen of Death?
Kids don't even know this. Lucky them.
GitHub is the new Internet Explorer 6. A Microsoft product so dominant in its category that it's going to hold everyone back for years to come.
Just when open source development has to deal with the biggest shift in years and maintainers need a tool that will help them fight the AI slop and maintain the software quality, GitHub not only can't keep up with the new requirements, they struggle to keep their product running reliably.
Paying customers will start moving off to GitLab and other alternatives, but GitHub is so dominant in open source that maintainers won't move anywhere, they'll just keep burning out more than before.
Github's two biggest selling points were its feature set (Pull Requests, Actions) and its reliability.
With the latter no longer a thing, and with so many other people building on Github's innovations, I'm starting to seriously consider alternatives. Not something I would have said in the past, but when Github's outages start to seriously affect my ability to do my own work, I can no longer justify continuing to use them.
Github needs to get its shit together. You can draw a pretty clear line between Microsoft deciding it was all in on AI and the decline in Github's service quality. So I would argue that for Github to gets its shit back together, it needs to ditch the AI and focus on high quality engineering.
> Monday
Beyond a meme at this point
I bet Microsoft did this...
Maybe we should post when it's up
it's Monday therefore Github is down.
GitHub has a long history of being extremely unstable. They were down all the time, much like recently, several years ago. They seemed to stabilize quite a bit around the MS acquisition era, and now seem to be returning to their old instability patterns.
Copilot, what have you done again?
presumably slophub's now dogfooding GitHub Agentic Workflows?
They should have just scaled a proper Rails monolith instead of this React, Java whatever mixed mess. But hey probably Microslop is vibecoding everything to Rust now!
Team is doing resume driven development
Is it just me, or are critical services like GitHub, AWS, Google, etc., down more often than they used to be these days?
3 incidents in feb already lmao
It's really pathetic for however many trillions MSFT is valued.
If we had a government worth anything, they ought to pass a law that other competitors be provided mirror APIs so that the entire world isn't shut off from source code for a day. We're just asking for a world wide disaster.
I get the feeling that most of these GitHub downtimes are during US working hours, since I don't remember being impacted them during work. Only noticed it now as I was looking up a repo on my free time.
Good thing we have LLM agents now. Before this kind of behavior was tolerable. Now it's pretty easy to switch over to using other providers. The threat of "but it will take them a lot of effort to switch to someone else" is getting less and less every day.
Are we sure LLM agents aren't the cause of these increasing outages?
tangled is up B]
migrating to azure kills businesses
Now Github pages are down
Welcome to Microsoft Github
GitHub downtime is going from once a month (unacceptable) to twice a month (what the fuck?)
[dead]
The next name after Cloudflare
That pink "Unicorn!" joke is something that should be reconsidered. When your services are down you're probably causing a lot of people a lot of stress ; I don't think it's the time to be cute and funny about it.
EDIT: my bad, seems to be their server's name.
I don't know if it's meant to be a joke, per se. They use (or used) the Unicorn server once upon a time:
I don't think it's a joke, it's the server that github runs on
I am personally totally fine with it but I see your point. Github is a bit too big for often braking with a cutsey error message even if it is a reference to their web server.
Even if it is their server name, I completely agree with your point. The image is not appropriate when your multi-billion revenue service is yet again failing to meet even a basic level of reliability, preventing people from doing their jobs and generally causing stress and bad feeling all round.
One of Reddit's cutesy error pages (presumably for Internal Server Error is similar) is an illustration that says "You broke reddit". I know it's a joke, but have wondered what effect that might have on a particularly anxiety-prone person who takes it literally and thinks they've done something that's taken the site down and inconvenienced millions of other people. Seems a bit dodgy for a mainstream site to assume all of its users have the dev knowledge to identify a joking accusation.
That stupid "Aww, Snap!" message I think it's one of the browsers does.
Github lost at least one 9, if not two, since last year's "existential" migration to Azure.
I'm sympathetic to ops issues, and particularly sympathetic to ops issues that are caused by brain-dead corporate mandates, but you don't get to be an infrastructure company and have this uptime record.
It's extra galling that they advertise all the new buzzword laden AI pipeline features while the regular website and actions fail constantly. Academically I know that it's not the same people building those as fixing bugs and running infra, but the leadership is just clearly failing to properly steer the ship here.
Is there any reason why Github needs 99.99% uptime? You can continue working with your local repo.
Many teams work exclusively in GitHub (ticketing, boards, workflows, dev builds). People also have entire production build systems on GitHub. There's a lot more than git repo hosting.
As an example, Go build could fail anywhere if a dependency module from Github is not available.
Are you kidding? I need my code to pass CI, and get reviewed, so I can move on, otherwise the PRs just keep piling. You might as well say the lights could go out, you can do paperwork.
> otherwise the PRs just keep piling
Good news! You can't create new PRs right now anyway, so they won't pile.
Lots of teams embraced actions to run their CI/CD, and GitHub reviews as part of their merge process. And copilot. Basically their SOC2 (or whatever) says they have to use GitHub.
I’m guessing they’re regretting it.
The Linux kernel uses an email based workflow. You can digitally sign email and add it to an immutable store that can be reviewed.
The money i pay them is the reason
What if you need to deploy to production urgently...
They didn't migrate yet.
Fucking REALLY?!
That's probably partly why things have got increasingly flaky - until they finish there'll be constant background cognitive load and surface area for bugs from the fact everything (especially the data) is half-migrated
This has those Hotmail migration vibes off the early 2000s.
Yeah, I'm literally looking at GitLab's "Migrate from GitHub" page on their docs site right now. If there's a way to import issues and projects I could be sold.
Maybe it's be reasonable to script using the glab and gh clis? I've never tried anything like that, but I regularly use the glab cli and it's pretty comprehensive.
I viscerally dislike github so much at this point. I don't know how how they come back from this. Major opportunity for competitor here to come around and with ai native features like context versioning