Forgejo Security Release 1.20.3-0

Today Forgejo v1.20.3-0 was released.

This release includes safeguards in case [storage].PATH is set or conflicting storage sections are found in the app.ini file. For instance if both [storage.packages] and [packages] exist, the directory in which the packages are stored may change after the upgrade. Forgejo will refuse to upgrade from v1.20.2-0 (or an earlier version) if the sanity checks fail and require manual intervention to avoid data loss, as described below.

It also contains a security fix that prevents leaking emails via the API.

Technical details of these bug fixes are available in the release notes.

UPDATE: this blog post was updated early September 2023 for Forgejo instances using S3 for storage. If you have read the previous version, please take a look at the diff.

We recommend that all installations running a version affected by the issues described below are upgraded to the latest version as soon as possible.

A manual action is required to avoid the risk of losing data if:

  • the app.ini file contains one or more [storage*] sections that are as follows:
    • [storage].PATH is set
    • [attachment] and [storage.attachments] exist
    • [lfs] and [storage.lfs] exist
    • [avatar] and [storage.avatars] exist
    • [repo-avatar] and [storage.repo-avatars] exist
    • [repo-archive] and [storage.repo-archive] exist
    • [packages] and [storage.packages] exist
  • the app.ini file contains STORAGE_TYPE = minio
  • you are currently currently running:
    • A Forgejo version lower than v1.20.3-0
    • A Gitea version lower than v1.21

If this is not the case this chapter does not concern you and can be skipped.

Bug description

The storage configuration in the app.ini file was refactored in v1.20 and bugs were introduced. There were also bugs in the previous implementation and all versions up to Forgejo v1.20.2-0 are impacted.

These bugs are best explained through an example. By default the files for each subsystems - Attachments, LFS, Avatars, Repository avatars, Repository archives, Packages - are stored in a dedicated directory. For instance if APP_DATA_PATH is set to /data, the directory layout looks like this:

/data/attachments
/data/lfs
/data/avatars
/data/repo-avatars
/data/repo-archive
/data/packages

But if the app.ini file contains the following section and no other storage related sections:

[storage]
PATH = /my/storage

all subsystems will share /my/storage instead of having their own directory. The attachments will be stored in the /my/storage directory, together with the avatars, the repository archives, etc.

Bug impact

Subsystems sharing a directory

If [storage].PATH exists in the app.ini file it may cause some subsystems - Attachments, LFS, Avatars, Repository avatars, Repository archives, Packages - to share the same directory.

It may not create a problem immediately and can go unnoticed for an extended period of time. But since each subsystem was designed to have a dedicated directory it will eventually:

  • Create a name clash when one subsystem tries to use the same files as another subsystem
  • Permanently destroy data when one subsystems delete files from another subsystem

One example of permanent data loss is when clicking on Delete all repositories’ archives (ZIP, TAR.GZ, etc..) in the system administration web page: it will not only delete the archives but also attachments, LFS files etc.

Misplaced data

When two sections related to a subsystem are found in the app.ini file (for instance [packages] and [storage.packages]), the location in which the data is stored may change in an unpredictable way after the upgrade to v1.20.3-0.

Prior to Forgejo v1.20, using STORAGE_TYPE = minio in some contexts was ignored and the files were actually found in local storage (for instance the [repo-archive] section as demonstrated by this test).

The Forgejo instance will no longer find the files stored in the previous location and it will start populating the new location. As time passes there will be no way to reconcile the content of the two locations that diverged in this way.

Relative paths inconsistencies

When the path to a local storage location does not start with a /, it is interpreted to be relative to another path. Unfortunately the logic changed across versions, as demonstrated by an extensive set of tests going back to Forgejo v1.18.5-0.

For instance if app.ini contains [storage.lfs].PATH = somedir it will end up in:

  • APP_DATA_PATH/lfs for Forgejo v1.19.4-0
  • WORK_PATH/somedir for Forgejo v1.20.2-0
  • APP_DATA_PATH/somedir for Forgejo v1.20.3-0

Forgejo supports two storage backends: the file system (the default or when STORAGE_TYPE = local is set) and S3 compatible storage (when STORAGE_TYPE = minio is set). To figure out where each subsystem stores its files before the upgrade, grep the Forgejo logs as shown in the following example:

$ grep -e 'New.*Storage()' -e 'Initialising.*storage' < forgejo.log
:initAttachments() [I] Initialising Attachment storage with type:
NewLocalStorage() [I] Creating new Local Storage at /data/gitea/attachments
:initAvatars() [I] Initialising Avatar storage with type:
NewLocalStorage() [I] Creating new Local Storage at /data/gitea/avatars
:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-avatars
:initLFS() [I] Initialising LFS storage with type: minio
NewMinioStorage() [I] Creating Minio storage at 127.0.0.1:9000:forgejo with base path lfs/
:initRepoArchives() [I] Initialising Repository Archive storage with type:
NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-archive
:initPackages() [I] Initialising Packages storage with type:
NewLocalStorage() [I] Creating new Local Storage at /data/gitea/packages
:initActions() [I] Initialising Actions storage with type: minio
NewMinioStorage() [I] Creating Minio storage at 127.0.0.1:9000:forgejo with base path actions_log/
:initActions() [I] Initialising ActionsArtifacts storage with type: minio
NewMinioStorage() [I] Creating Minio storage at 127.0.0.1:9000:forgejo with base path actions_artifacts/

NOTE: when a Forgejo container is configured using variables such as -e FORGEJO__storage__PATH=/my/storage, it will create an app.ini file that contains [storage].PATH. Removing this variable will not remove the section from the app.ini file, it has to be done manually.

Follow the instructions below to update your app.ini so that the storage locations stay the same. Keep in mind that some manifestations of these bugs are not covered by the sanity checks preventing an upgrade that may have an impact on the storage location. For instance when relative paths are used in the app.ini file or other corner cases that have not been discovered yet.

If you have any doubt about the following steps or if you suspect the storage directories are merged together as described above, file an issue or reach out to the chatroom to get help.

  • Before upgrading to Forgejo v1.20.3-0

  • Save the output of grep -e 'New.*Storage()' -e 'Initialising.*storage'

  • For each local storage, add a section to app.ini, replacing the PATH value with the absolute path of the directory for each subsystem. Check the table in the documentation to find the correspondance between the name of the subsystem displayed in the logs and the section in the app.ini file. For instance, the subsystem Attachment is associated with the [attachment] section:

    [attachment]
PATH = /my/storage/attachments

  • For each minio storage, add the following to app.ini, replacing the MINIO_BASE_PATH value with the base path found in the logs (see the grep example above). Check the table in the documentation for a correspondance between the name of the subsystem displayed in the logs and the section in the app.ini file. For instance, LFS needs:

    [lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = mylfs/

MINIO_ENDPOINT = 127.0.0.1:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1

  • Remove the [storage] section from app.ini

  • Remove the [server].LFS_CONTENT_PATH entry from app.ini (it is the default for [lfs].PATH)

  • Remove the [picture].AVATAR_UPLOAD_PATH entry from app.ini (it is the default for [avatar].PATH)

  • Remove the [picture].REPOSITORY_AVATAR_UPLOAD_PATH entry from app.ini (it is the default for [repo-avatar].PATH)

  • Merge the settings found in related sections together as follow:

    • move the settings found in the [storage.attachments] section into the [attachment] section and remove it (one plural, the other singular)
    • move the settings found in the [storage.lfs] section into the [lfs] section and remove it
    • move the settings found in the [storage.avatars] section into the [avatar] section and remove it (one plural, the other singular)
    • move the settings found in the [storage.repo-avatars] section into the [repo-avatar] section and remove it (one plural, the other singular)
    • move the settings found in the [storage.repo-archive] section into the [repo-archive] section and remove it (both singular)
    • move the settings found in the [storage.packages] section into the [packages] section and remove it (both plural)

  • Upgrade to v1.20.3-0 or a later version

  • Verify each subsystem uses the expected storage with grep -e 'New.*Storage()' -e 'Initialising.*storage

Bug fix and data recovery

After upgrading to v1.20.3-0 the storage settings can be used as explained in the documentation. But there is unfortunately no way to automatically repair an existing instance impacted by these bugs during the upgrade.

To address this problem:

  • v1.20.3-0 and later will refuse to upgrade from v1.20.2-0 (or an earlier version) if the sanity checks fail. Upgrading without manual verification would be taking the risk of silently changing the location in which a subsystem expects to find its files. The data would not be lost because it would still exist in the former location, but Forgejo will not find it any more. The new location will start being populated in a way that may be impossible to reconcile with the content of the former location.
  • v1.20 point releases from v1.20.3-0 and later will refuse to downgrade to v1.20.1-0 or v1.20.2-0. Although it is usually possible to downgrade from a point release to a lower point release, it is forbidden in this case to protect the Forgejo instance from any risk of data loss.

NOTE: A recovery strategy to separate directories that were previously merged together can be to duplicate the merged data into the target directories. There can still be name clashes and it only works for local storage if the amount of data is not too large.

Get Forgejo v1.20

See the download page for instructions on how to install Forgejo, and read the release notes for more information.

Upgrading

Carefully read the breaking changes section of the release notes.

The actual upgrade process is as simple as replacing the binary or container image with the corresponding Forgejo binary or container image. If you’re using the container images, you can use the 1.20 tag to stay up to date with the latest 1.20.x point release automatically.

Make sure to check the Forgejo upgrade documentation for recommendations on how to properly backup your instance before the upgrade. It also covers upgrading from Gitea, as far back as version 1.2.0. Forgejo includes all of Gitea v1.20.

Contribute to Forgejo

If you have any feedback or suggestions for Forgejo do not hold back, it is also your project. Open an issue in the issue tracker for feature requests or bug reports, reach out on the Fediverse, or drop into the Matrix space (main chat room) and say hi!