500/504 server error creating issues/PR, commenting etc. #2596
Labels
No labels
accessibility
bug
bug
infrastructure
Codeberg
contributions welcome
docs
duplicate
enhancement
infrastructure
legal
licence / ToS
please chill
we are volunteers
public relations
question
question
user support
s/Forgejo
s/Forgejo/migration
s/Pages
s/Weblate
s/Woodpecker
security
service
upstream
wontfix
No milestone
No assignees
54 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Codeberg/Community#2596
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Comment
It had been happening several hours a day for previous 3 days.
E.g. when posting a comment a "loading" (sending?) spinner is visible and it spins for several seconds, then a red
! server error: 500error bar appears and upon page refresh its clear the comment hasn't been added.It also happens when e.g. rebasing a PR and force pushing changes - the code gets updated, but the force pushed message don't get recorded. Same with merging a PR - it gets merged, but its status is not updated to "merged" etc.
First time it had been resolved I thought its a transient problem that had been fixed.
But now its happening again for a third time..
I.e. I was not able to create this particular issue at ~9:20am UTC, so I've reported to the matrix chat and waited until the issue is gone again.
(CC @gusted)
A general comment on the deadlock issues, yesterday during Codeberg's weekly community meeting we did look into it and discovered one of our Galera (MariaDB database cluster) node is slowing down the whole cluster. The underlying cause is not confirmed, we saw troubling numbers on one of the root drive SSDs and the machine of that node has the weakest CPU. This machine also hosts most services (notable exception being Forgejo and Woodpecker CI agent) after it was moved to this server when our third machine suddenly died (https://social.anoxinon.de/@codebergstatus/115974907704158246), so finally moving it back is also likely planned to distribute the CPU load.
For now we've increased the amount of threads (
Codeberg-Infrastructure/scripted-configuration@d80ce8c6c2) and is showing promising results, although I do still see codeberg.org returning 5xx errors when the operation (mostly) succeeded so the issue is not completely gone.There's also a possibility it being due to a Forgejo v15 upgrade as we also did that quite recently, but so far nothing has been able to point towards a possible suspect in that area. We do see "weird"
DELETEqueries from time to time that the database struggles to process that might suggest Forgejo is sending too complex queries to the database.I'm also seeing the same issue - creating a PR, it just spins then gives me a big red 500. I've tried several times so far, but no success.
@jacklund I can see a few deadlocks around that time - unfortunately the last deadlock recorded by mariadb (for which I can see detailed information) was unrelated to commenting/issue/pr :(
Forgejo v15 gained the ability to retry transaction, I will add some retries around the areas where I can see deadlocks from around that time. -> Deployed as part of
Codeberg-Infrastructure/forgejo@7f55180a7dI was going to open an issue on behalf of the Gentoo PR assignment and CI with more or less the same title: Frequent 500 errors. From what I understand this appears to happen particularly often around deleting comments.
The scripts in use that interact with Codeberg are found here:
In both repos, the API managed in a
CodebergAPIclass defined in codebergapi.py. Both set of scripts are executed in a cron job, and post comments on PRs. They also clean up old comments, so there's somewhat frequent comment deletion.For example: https://gitweb.gentoo.org/proj/assign-pull-requests.git/tree/assign-pull-requests-codeberg.py#n175
An example PR: gentoo/gentoo#853 - includes both "Pull Request assignment" and "Pull request CI report".
Deleting things are quite a thing, as they don't just delete the entry from the database but also delete references to it and then you're suddenly touching quite a few tables. And yeah that makes it more prone to deadlocks. That said, yet another function that can benefit from
RetryTxto make it (slightly) more robust; will add it for the next deployment.I cannot add a user to the team of a repo. (again 500 errors)
@jakorten your 500 is not related to this one, please check the username that you're entering (specifically remove some whitespaces at the beginning of your input)
Interesting, but now it does work (I copy-paste usernames so I doubt it has to do with spaces)
@jakorten Your input into the username field is
user-namethe trailing whitespace need to be removed.This might now manifest as 504 errors: https://social.anoxinon.de/@codebergstatus/116540639518944405
We've identified a regression in Forgejo. We were able to catch a problematic query that was locking the
commenttable for several minutes, causing all commenting related operations to deadlock. Query itself was a simpleDELETE FROM comment WHERE dependent_issue_id IN (list of issue ids),dependent_issue_idcolumns has a index. This should be a very fast operation, but upon manually executing the query with some test data it was really slow, theANALYZEquery provided a very clear reason: index was not being used. Trial-and-error shows the limit was 200, a value which happens to be the default value of eq_range_index_dive_limit.Here's where the regression comes in, the length of the issue ids list was previously 50 but is now 500 with forgejo/forgejo#11999, therefore going over that limit and hitting this optimizer. Why this optimizer is removing the use of the index is still unknown, and possible a bug, we are able to resolve the performance issue by bumping the threshold to 501 to restore to the old behavior of not using this optimizer.
Codeberg-Infrastructure/scripted-configuration@4800a045b1I will keep this thread open for a few more days, but I believe this eliminates most of the deadlock errors we've been seeing in the past few weeks.
I still get 504 on repo creation
@breinich wrote in #2596 (comment):
I have the same problem. Both when migrating and when creating a new repo.
getting 504 while trying delete a repo
I experienced this error these days (also today [1]) which is specially noticeable at 12:00-13:00, on other day moments, like now, works better (specially at night).
I am thinking, what about a test/healthcheck that creates an issue each hour, evaluates time (or timeout), I suspect that it's because of heavy-balance, it might need require extra resources on "prime time", but maybe, there is more optimization to address that heavyness.
example/inspiration: https://matrix.org/blog/2020/11/03/how-we-fixed-synapse-s-scalability/#performance
[1] this was creating an issue, trying different moments (I got rate limited pretty fast in a previous attempt, so this time, I was more relaxed on retrying for several minutes)
Currently cannot create a repository, I keep getting hammered with a 504.
I still get the same thing as above.
Yes, the 504 error is back. I can't star a repo and I can't create a new repo. Git operations seem to be working fine.
I also can't create repo's and subscribe to threads
I also cannot create a repository and am getting the 504 error.
This is fixed now. I just created a repo and starred another repo.
Seems to work for me too!
This has unfortunately manifested a few more time since we found the regression and applied a workaround to it. In the incidents were I was around and able to observe the 504 they still stemmed from the same query that we identified to be problematic in #2596 (comment) and killing it (or restarting Forgejo) was the only way to unblock other queries. In the hopes it does something
Codeberg-Infrastructure/forgejo@6d221141a3was deployed to restore the old behavior of smaller batch sizes.We're still going to look more into it, but unfortunately the time where we can diagnose, test and troubleshooting the issue is when certain operations are not working (and you get the 504 gateway timeout). This does not give us a lot of time to properly deal with it and have to end those sessions early to simply restore functionality.
Still experiencing this when trying to create a PR (red error bar
! server error: 500).500 server error creating issues/PR, commenting etc.to 500/504 server error creating issues/PR, commenting etc.Unable to fork ziglang/zig. Consistently seeing 504 errors.
Getting 504 when trying to create a repository too. Seems like this issue lasts since last week. Damn
Looks like I can't make new issues.
I'm consistently getting 504s when creating repositories, hope this gets resolved soon! <3
Edit; Staring and watching repositories also results in the same error.
Can confirm that I am unable to create new repositories
I'm also experiencing 504s. Clicking "watch" on this issue returns 504, and attempting to create a new issue is also returning 504
Failed to star or fork any repos, too, the error code is 504. Any updates on this issue?
Error 504 when calling /Codeberg/Community/issues/2596/watch: <html>
504 Gateway Time-out
The server didn't respond in time. </html> :(Same here, can't create new repositories
Same issue when trying to create or delete a repository.
I've been trying to make a fork for a PR for the last 5 hours
I also can't add stars, watch projects, or follow issues (I discovered this while trying to follow this one).
I get the same issues
Same here, been trying to create a repo for I'd say the last couple of hours and cannot due to this.
Same issue, cannot delete or create a repo. Cannot star or unwatch a repo either. RSS button works fine though.
As many others I get 504 errors when trying to fork a repository or subscribe to some issue. How can we help from a user perspective? Can we collect and provide any data to help in troubleshooting?
I think the status.codeberg.org website should reflect these issues; I honestly thought it was a problem with my provider or me because everything looked fine on the status website.
It's been resolved for now.
@calsan This can't be integrated into the status page easily, it only affects certain endpoints and as they effect write operations the status page can't replicate that as a health check.
@Gusted I understand, I was just saying that because I recently migrated from GitHub and I figured it's very frustrating for a new user to see that something they need isn't working when there's no information about it. I mean, obviously this issue exists, but if you check the status page first, I mean, it took me four hours before I found it. I was trying with VPNs or my third-world providers. Anyway, I know it's not your or the team fault; in fact, thanks, it seems to be working now.
To also give a status update on the underlying problem. Whenever a big (many issues, comments, pull requests, action runs) repository is being deleted this is done in a transaction, and at some point it effectively has locked most tables and is blocking other INSERT/UPDATE queries to the point those are timing out. I'll be prioritizing to reduce the "harm" that this transaction can do, but it's no easy task and requires some non-trivial engineering. If you've ideas please reach out in https://matrix.to/#/#forgejo-development:matrix.org
I'm adding that I'm seeing 504s when trying to fork a repo.
[EDIT: Tried again just now, ~4 hours later. Worked fine.]
same here trying to fork guix/guix for two days, without success.
I'm responded with a 504 after 30s delay.
Is this like an intermittent issue surfaced during high load, or should we expect for the time being that forking a large repository is going to trigger a timeout? Not trying to pile on, just looking for a little more clarity on the current assessment. ty.
@Gusted gave an explanation on the underlying problem yesterday.
TL;DR: When you try to fork some repo while the database tables are locked you running into a timeout.
I have same 504 error on simple attempt to add comment here
freesewing/freesewing#848
Whello.
We too are still experiencing it.
some logs:
https://github.com/cinepro-org/core/actions/runs/26169397922/job/76983309056
https://github.com/cinepro-org/docs/actions/runs/26169524670/job/76982973737
I hope you can fix it soon.
A little bit funny that I got a 504 trying to open this issue.
When the issue tracker is affected, you just know it is bad ...
Clicking on one of the larger (< 1Mb) files in my repo results in 504 error every time.
works again....
still not work...
I'm also running into this when trying to change my username.
(edit: worked almost immediately after posting this comment...)
This whole situation is such a mess. Sometimes codeberg does work, other times it doesn't. I wish someone could give a clear explanation of why this happened and how and when it will be resolved.
@thomasboom wrote in #2596 (comment):
See the following two comments for an explanation and what the team behind Codeberg.org is currently doing about it:
There are a lot of transient 504s, but in plenty of cases the problem can be reproduced 100% of the time.
I've been trying to fork forgejo/forgejo for almost a week now, and every day I get a 504. I have yet to succeed.
I don't know if that is the same root cause as all the other 504s reported in this thread, but surely there is no problem with reproducing this particular issue?
@Tronde wrote in #2596 (comment):
While I completely understand the volunteer nature of Codeberg, comment 2 does not give us a lot to work with:
Ok.. I get that complex ephemeral bugs are hard to diagnose. On the other hand.. it's literally making Codeberg unusable. Like, my team can't create ANY PRs. That's not a sometimes thing, it's not exotic behavior.. it's literally the main day-to-day task.
It's not intermittent, either -- it fails every single time with a
500error.(We've had to switch away from Codeberg until this is fixed.)
It also sounds like, for whatever reason, this issue is mixing together at least two distinct class of errors -- the
504timeout, and the plain500error on creating PRs.It doesn't sound like the latter is related?
It would be great to read a statement (or follow-up) at some point on whether this issue is mainly due to the use of old/low-powered hardware (in relation to the load the instance has meanwhile), given that a general bug in Forgejo has been outruled by now?
The issue described in #2596 (comment) was fixed yesterday in the morning. The results so far are good, and we believe the main source of 504 issues are hereby resolved that were related with: creating issues, comments, pull requests, starring and watching (repo-related actions that touches the database with a INSERT or UPDATE statement).
Forking is a different issue, for large repositories there's a slow process of migrating tags which takes ages. We already raised this timeout to 10minutes but I just noticed there was a small ordering mistake where this didn't get applied correctly. @untitaker
@pat-s, it's solely Forgejo being put under stress by the amount of traffic we're seeing and if that's combined with Forgejo making some assumptions that fail under scale, that's not fixable by better hardware. Also sorry of the mess of this thread but I hope I didn't give the indication Forgejo was outruled, it was the prime suspect 😄
@codenamedmitri Sorry it seems my comment was poorly phrased to convey its intent, it's not a ephemeral bug. Rather once we're aware of it we also have in the back in our heads that we should resolve it ASAP because it's affecting people's workflow by the second as you've noticed. If we then want to casually read documentation, what tables are good to be used to diagnose etc. then that becomes quite stressful and is better left for the next attempt and resolve it temporarily by stopping the offending query.
@bagel-very your case deserves a new issue. Although we today deploy a general bump to 30s as a trade-off, so maybe that resolves it.
I have CI (GitHub workflows) that have been getting 504 intermittently in the last few hours.
Example workflow log:
https://github.com/owncloud/core/actions/runs/26386534375/job/77666167782?pr=41552
And again:
https://github.com/owncloud/core/actions/runs/26388508278/job/77672390895?pr=41552
Hi, if you saw any 504 or 500 in the last ~10 hours then it was related to someone trying to bring the instance down.
Having 504 issues in GitLab CI's of Fdroid repo: https://gitlab.com/albertodiazsaez/fdroiddata/-/jobs/14550532584
Don’t forget to check https://status.codeberg.eu/status/codeberg when there are such issues. Seems again #2596 (comment).
@mahlzahn wrote in #2596 (comment):
Status showed All Systems Operational, and still the pipelines are failing with a 504, I guess it's the same issue as you mentioned, but the Status page won't show it sometimes. Website seems to work, I only detect problems when working with git commands.
I've been getting 504s while fetching my repo over https and connection closed with ssh since this morning. Both of my machines are affected so I don't think its a config issue on my end.
The latest 504s are being caused by a load issue, we're not really sure where it's coming from but it seems related to incoming SSH connections.
same orphaned *.lock under /mnt/ceph-cluster on push, 504 at finalize — details in #2707, also on niko64/vecgfx #2710.
Another occurrence, this time on
git push(not issue/PR creation): repoAnarBib/anarbib. Objects transfer fully, then the push dies at the post-receive / ref-update step:Writing objects: 100% (56/56), done.
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
The server-side ref never updates (web UI still shows the ~24h-old HEAD). A 504 at this same finalize step earlier left a stale
refs/heads/main.lock(since cleared — thanks). I'm holding off on retrying to avoid recreating a lock. Details in #2707. Happy to provide timestamps.From Gentoo's CI perspective: It's working well the majority of the time, we sometimes see a 504 but they're not that frequent. Mostly it's this error we see in the logs (fetching the list of open PRs):
@Gusted wrote in #2596 (comment):
Sorry if off topic. But would we be able to convert this statement into a feature request? I have two options for this feature request in mind.
MattTheTekie referenced this issue2026-06-09 22:06:16 +02:00
@laumann there's a big performance problem with that endpoint that makes it unnecessary slow, we already had to give a exception for it to Forgejo project (
http-request set-timeout server 60s if { path /api/v1/repos/forgejo/forgejo/pulls } { url_param(limit) eq 100 }) I will add one for gentoo as well.
@kaeru feature requests are better discussed and proposed in Forgejo: https://codeberg.org/forgejo/forgejo/issues/new/choose
@Gusted wrote in #2596 (comment):
Thanks, that's very nice of you :) let me know if there's any changes we could be making to alleviate performance issues.
@Gusted is there any chance to fix this for users? I am unable to create pull requests to my own projects which renders codeberg unusable at all.
Currently getting this error when I try to pugh:
fatal: unable to access 'https://codeberg.org/FlynnD273/pebble-nomai.git/': The requested URL returned error: 504Same:
fatal: unable to access 'https://codeberg.org/lambert63/BFR.git/': The requested URL returned error: 503
Seems to be solved. I can now push to my repo's.
The current errors do not seem to stem from any systemic issue. It's merely occasional hiccups from what I'm observing, which is a always on-going effort to improve the overall reliability of codeberg.org
@Gusted still unable to create PRs, either programmatically or manually through a browser. Get 500 error all the time.
Experience this since yesterday... I had created one PR and several hours after I can't...
I count seven different issues in the last 24 hours that have been opened about 504 errors/repo locks blocking usage of repositories. #2880 #2882 #2884 #2885 #2886 #2887 #2889 . I posted in the matrix chat yesterday and this morning and have received no response of any kind on my ticket or in the matrix chat. I would appreciate at least knowing that someone is working on a fix, since currently it is blocking me from doing any development effectively. Our only workaround is to temporarily switch back to github until this issue is fixed because it is a complete showstopper, and I would much rather not do that since I prefer and appreciate codeberg. Any information on what the issue is or if a resolution will be forthcoming would be appreciated immensely.
Edit: Gusted let me know via Matrix that they are currently working on this issue and have been since yesterday. Thanks again to the people who volunteer their time to help keep Codeberg operational.
the same
I'm unable to fork fediverse/fep - I've ended up with a bunch of orphaned repos, where if I try to fork it again to the same name I get "Files already exist for this repository. Contact the system administrator." Otherwise, it just hangs reloading.
Update: it's back for me.
While interface gives me 500 error when I try to create PR via web, AGit workflow allowed me to create PR
Another downtime in the last few hours, constant 503.
There are no incidents logged on https://status.codeberg.org/status/codeberg in July, but there have been plenty of issues for time-to-time.
Why are these things not logged/reported as incidents, so that we know what has been going on?
I'm on the fence to close this issue as it no longer reflects why it was opened in the first place: (1) individual incidents that are not seen/reproduced by other people (for this a new issue would be the best way to track it so it can be looked at by a Codeberg admin) (2) incidents that are already being tracked/aware of by Codeberg via https://social.anoxinon.de/@codebergstatus/
@holtowd @abbra please open a new issue, it's quite likely something else than the outage/downtime of Codeberg.
@phil-davis we no longer uses the incidents of uptime kuma, this was reported in the may 5 incident. Unfortunately it seems someone still added recent incidents and this message is now hidden beneath the incidents. The primary source is https://social.anoxinon.de/@codebergstatus/ as this allows for more updates on specific incidents over time via the use of threads.