Forgejo monthly update - October 2024

The monthly report is meant to provide a good overview of what has changed in Forgejo in the past month. If you would like to help, please get in touch in the chatroom or participate in the ongoing discussions.

Two years - a recap

In October 2022 Forgejo was announced in reaction to the takeover of Gitea. After a two months of preparation, the first release was published and Codeberg announced using it because “it provides Codeberg with an essential feature: trust”.

The security team got quite busy soon and published multiple releases. The release team was also able to deliver but a mistake was made. This was the first occasion for Forgejo to show that problems are explained transparently and their impact articulated clearly. The integrated CI, Forgejo Actions, was announced and started to be used by Forgejo itself very early on.

In February 2023 someone new (who wasn’t a contributor that the project is relying on) joined the chat and issue tracker, spoke repeatedly in ways that was hurtful/painful to Forgejo community members, and did not seem to have capacity to speak more sensitively, despite offers for support and repeated requests. It distracted community members from productive and important work on governance, strategy and development. Some community members went silent, others were on edge. The moderation process was created during these challenging times. It took months for the community to heal.

In search for long term sustainability, the first grant application was sent. It was awarded and the funds allowed Codeberg to hire developers early 2024. It was not perfectly managed and in December 2024 a significant part of the funds will be returned because they were not spent. It currently is the priority of the sustainability team established in August 2024.

After weeks of discussions, a decision was made to welcome copyleft contributions in Forgejo in June 2023. The Forgejo decision making process requires that all concerns are heard and answered before a decision is final. It takes long but is also a key to being inclusive. It became a reality a year later, in August 2024.

Forgejo federation is and will always be the highest priority of the Forgejo project. Every month, since the beginning, updates on its progress are published. It is still not in a usable state, two years later, and that has caused some frustration but the work continues.

In August 2023 a regression was discovered to cause data loss. A lot of work went into fixing it and publishing documentation explaining how to recover. It was caused by a refactor that was not properly tested and was one of the main motivation to require that every pull request merged in Forgejo is tested.

In the last months of 2023, Forgejo contributors kept improving while rebasing all the changes on top of the Gitea codebase. However, when Gitea Cloud was announced in December 2023 and after some investigation, it became clear that Gitea turned Open Core.

In January 2024, the Forgejo localization team came into existence, in anticipation of a hard fork. Before that, the Forgejo translations depended on Gitea translations which are trapped in a proprietary service. The initial localization team covered Arabic, Dutch, French, Russian, Greek and German and kept growing since.

Forgejo was ready for such an event and declared its intention to become a hard fork, separating itself from Gitea even further. Just as for the decision to welcome copyleft contributions, this required weeks of (sometime intense) discussions. And it also took weeks of work to be implemented in March 2023. Coincidentally the Open Core turn of Gitea was confirmed when a the first proprietary version of Gitea was announced around the same time.

There was a sense of liberation when the hard fork began: it was possible to write code incompatible with the Gitea codebase! But there was also a price to pay: features and bug fixes relying on such code could not be shared with Gitea. It would have been easy to be carried away and get stuck with not enough contributors to maintain a codebase that diverged too quickly. To mitigate that risk dependency management tooling and a weekly observation of Gitea activity was organized and is still in place.

The Forgejo v9.0 release that was published in October 2024 is the third major release after the hard fork. It includes a feature that would have never been possible before (quotas) because it requires architectural changes conflicting with the Gitea codebase. Forgejo v7 is a Long Term Support release, the first of its kind, supported during a year instead of three months. It is another benefit of the hard fork, made possible because Forgejo is no longer bound to the Gitea release cycle.

In these past two years Forgejo matured and transformed into an independent project, with a solid user base and a lively community of contributors. It involved a lot of coding and other time consuming technical work. But it was first and foremost a human adventure, with its share of plot twists and drama.

Forgejo releases

On 16 October Forgejo v9.0 was published. It is the first version to be released under a copyleft license. Codeberg was upgraded a week later. Regressions were discovered and fixed. Some of them were only noticeable visually (diagrams not showing labels or the displayed name of archives). Another was about the container image size that grew significantly (180MB for v9.0.0) and was reduced to 70MB for v9.0.1, back to the size of the Forgejo v7 images.

On 28 October Forgejo v9.0.1 was published and fixes those regressions. It also contains two security fixes that were backported and published as Forgejo v7.0.10, the Long Term Support version.

These releases are the first to reflect the new Forgejo lifecycle. Before Long Term Release support existed, only v9 and v8 would have been supported, i.e. the last two major versions. But since v7 is supported until July 2025, the supported versions are now v9 and v7, i.e. the latest version and the long term support version.

User Research

The user research team conducted a survey regarding the repository settings during two weeks in October. It encouraged participation by showing banners to users of Codeberg in the repository section, linking to an external survey on Cryptpad.

There have been 118 submissions and the analysis is still ongoing, but there is already valuable feedback among the reviewed feedback. Thanks to all the participants!

There has been a rather spontaneous interview regarding accessibility with a Codeberg user. They reported a serious issue with their screen reader, which we didn’t yet manage to reproduce (even after a contributor set up a test environment with the proprietary operating system and the screen reader). Investigation of this issue currently has high priority and we hope to fix the issues as soon as possible.

Security Policy

Forgejo published its security policy to clarify communication and collaboration of the Forgejo security team with external parties such as libraries, security researchers and users.

Advance notice of security releases are available publicly. They do not contain specific information until the day of the release and are meant to help Forgejo admin plan for an upgrade.

Gitea was given a detailed description of the security issues fixed in the the v9.0.1 and v7.0.10 releases in advance, as well as a patch waiving copyright to fix them. From now on, any third party willing to receive such details in advance is required to explicitly agree to comply with the security policy.

Helm chart

A new major version, 10.0.0 was published. It updates the Forgejo docker tag to v9.

The Forgejo helm chart had security patch updates, in both v7 and v10. Helm chart v7.1.3 and v10.0.1 are the latest.

Localization

The translation hackathon (translathon) organized by Codeberg this month resulted in many new contributors joining and making thousands of additions and improvements.

In total, 57 people contributed to the translations this month, which is significantly more than any previous month.

A new script was added to process the localization files and verify that they contain only valid HTML insertions that follow the strictly defined rules. This should make it nearly impossible to insert malicious HTML.

Due to project’s legacy, the localization strings traditionally were able to contain any arbitrary HTML code and often had hardcoded links and other aging code. The addition of this script reduces the number of attack vectors on Forgejo’s codebase and improves it’s maintainability. Fortunately, there have been no security incidents caused by this flaw.

Infrastructure

A new k8s cluster was created and planned to replace the current setup. Instead of ad-hoc scripts, conventions and associated documentation, it relies on a declarative description that updates the cluster when a commit is pushed to the repository.

It went through a few disaster recovery tests and is now in production, hosting https://next.forgejo.org and https://v7.next.forgejo.org, ready to welcome other Forgejo instances.

The motivation for creating this new cluster is to improve the availability of https://code.forgejo.org in the wake of last month downtime. But it also significantly improves automation and reduces the technical debt. It will obsolete the ad-hoc scripts (wakeup-on-logs, shell scripts, …), conventions and documentation.

A k8s cluster is more attractive to Forgejo contributors who are willing to improve and maintain the infrastructure. They are in familiar territory if they already know k8s and do not need to learn new tools. They can start contributing with pull requests to the repository describing the cluster and eventually apply to become a member of the devops team when they gained enough trust.

It is a lot more work to learn k8s from scratch than it is to learn the current ad-hoc system from scratch. From that point of view, this transformation does not make it easier to find volunteers willing to participate. However, there are a lot of devops who already learned k8s while nobody knows the current ad-hoc system. They do not need to learn k8s and can jump right in.

Sustainability

The beneficiaries of the NLnet grant application sent in April 2024 are no longer available. A call for participation was posted to find Forgejo contributors willing to participate.

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is meant to reflect this diversity and acknowledge all contributions since the last monthly report was published. If you are missing, please ask for an update.

A minority of Forgejo contributors earn a living by implementing the roadmap co-created by the Forgejo community, see the sustainability repository for the details.