HomeInstallBlogExploreTeamDonate

Why and how we migrated from gitlab.com to gitlab.postmarketos.org

October 14, 202410 min. read

We have successfully migrated from gitlab.com to a self-hosted instance at gitlab.postmarketos.org! This was quite the journey, so here is a blog post looking back at the reasons, what went well and what didn't, and some lessons learned that might be useful for other free software projects considering a similar move. The reasoning section of this blog post is based on what we wrote earlier in #77.

Special thanks to OSUOSL for hosting this GitLab instance. It has been a pleasure working with them every step of the way!

Reasoning

Why move away from gitlab.com?

We are very grateful to the gitlab.com folks for hosting postmarketOS since 2018. They even gave us the "free premium features for cool open source projects" thing. However the main reason we wanted to move away from gitlab.com was that the requirements for signing up and using the site have changed dramatically over the last years.

While in 2018 when we migrated over to gitlab.com, there were no problems accessing the website and git repositories over for example Tor or setting up an account without giving any meaningful personal data. Nowadays, users of gitlab.com are required to provide a valid phone number and credit card information when setting up an account.

Also the website is protected by Cloudflare, blocking lots of proxies and sometimes even making the site hard to use for users with their real IP — event those who set an account up years ago and have been active since. It is very understandable that gitlab.com needs to take measures to avoid spam attacks and prevent misuse of their incredibly useful resources. But that doesn't change the fact that we've likely lost out on potential contributors due to these policies. Over and over again we heard from people who would have loved to contribute to postmarketOS, but have no interest to do so because gitlab.com requires too much of their personal information.

We highly appreciate that an Open Source Program Manager from GitLab reached out and asked us for our concerns. So far the situation has not really changed, but he confirmed that we are not the only community to face these challenges with GitLab, and he tried to make his team aware of these problems.

Who is maintaining the instance?

The new gitlab.postmarketos.org instance is maintained by OSUOSL, the Oregon State University Open Source Lab. They already provided various machines and services to free and open source projects, including Alpine Linux, Chimera Linux, Gentoo, GNOME, KDE, Kodi, OpenBSD just to pick a few from this huge list. Clayton talked to them at the FOSSY conference and they offered to host our GitLab instance too, and provide us with GitLab runners for x86_64 and aarch64 as well. We have decided that we will make a one time donation to them (TBD). You will be able to see it on our Open Collective page just like all our other spendings.

The benefit for us is that we have mostly the same code forge we have been using for years (we already tried to avoid using features that are not available in the self-hosted version of GitLab), but without the aforementioned GitLab.com-specific problems and without the need to invest time into maintaining the instance ourselves — time we would much rather spend on moving postmarketOS forward.

Why not SourceHut/Forgejo/...?

We think both SourceHut and Forgejo are amazing projects and if you are considering a move for your project, you should definitively take a closer look at them! They just weren't the right fit for us at this moment and we wanted to get it done.

SourceHut: we evaluated it as code forge for some time with pmbootstrap.git, but then moved it back. See the related announcement for details. postmarketOS has been using SourceHut Builds to build its binary packages and images since end of 2019 and this works great, we don't have plans to change that.

Forgejo: This is the Gitea fork that Codeberg uses. It is close to our requirements, but when we checked there were some details that held us back like not being able to have open discussions block merge requests / PRs. It would probably be a good fit if we invested more time and set up our own CI runners. But at this point we have been looking for an alternative to gitlab.com for two years already. We are happy that we have a solution with an easy path forward where we don't need to spend too much energy (that again we can't spend on improving postmarketOS).

Preparation

Announcement

OSUOSL was quick with preparing a GitLab instance for us. We configured DNS to to point gitlab.postmarketos.org to it. Then on 2024-08-25 we created our announcement #77 with:

We also announced it in the monthly blog post on the same day.

Test imports

After making some obvious changes such as uploading our logo, it was time to do a full test import. How long would it take to migrate our postmarketOS group? 30 minutes? Two hours? But first we found that there are actually two ways to import groups and projects.

File export vs. direct transfer

File export:

Direct Transfer:

So this was not great, and the related documentation is hard to read. Only after reading it several times, it now clear that "preserving user contributions" might have worked if users had set a public email and a user with the same address had already existed in the self-hosted instance. Most users of course don't have a public email configured and we don't want to ask people to do this just for this migration.

Trying "file export"

We tested "File export" for our biggest projects pmbootstrap and pmaports.

pmbootstrap:

pmaports:

Administrator, Administrator and Administrator reacted with 👍

Afterwards we were quite disappointed that "preserving user contributions" did not work as we expected. Instead of having the original usernames in issues and merge requests, comments, etc., all of them were replaced with the user that did the import. We did not set our emails to public, and looking back at it this might have made it work — but then again asking all of our users to set their email address to public to have everything imported correctly isn't practical.

As the direct transfer migration said that "an administrator can create users on the self-managed GitLab instance", we thought that this would be fixed if only we could use the direct transfer method. We decided to reach out to GitLab via the email address that was edited into the top post of the Direct Transfer status, dead for too long issue.

We were told to open an issue on https://about.gitlab.com/support and proceeded to do that. It was a bit awkward, because we had to fill an "emergency support request" as the premium for open source tier doesn't include any support. However the support people were nice and after some days of back and forth and waiting for other people at GitLab to make decisions, they asked for a gitlab.com account name and enabled the direct transfer feature for that account.

Meanwhile we announced that the migration would be done about a month later (later settling on the date 2024-10-06). We would either get it working with "direct transfer" until then, or just use the "file export" method with much more manual effort.

Trying "direct transfer"

Importing all postmarketOS projects took about 18 hours. It mostly worked, there were some errors regarding the pastebin-like feature that we don't use anyway and a custom emoji that could not be imported. But nothing important had failed. However the "preserving user contributions" did not work with this method here either, even though we did the direct transfer as administrator and at least one account did already exist in the selfhosted gitlab instance with the same email address and username! In hindsight, it might again have worked if we set the emails to public, but this isn't even mentioned in the "direct transfer" documentation.

We decided to go ahead without "preserving user contributions". At least the direct transfer allowed us to not manually export and import each and every project.

To be fair: from privacy perspective it makes a lot of sense that you cannot just export email addresses from all users that ever contributed / commented in your project without their consent. And just a few days after our migration we got an email (apparently along with everybody else who had "direct transfer" enabled at that time) stating that GitLab is working on an improved solution for user contribution and membership mapping that doesn't rely on emails. So hopefully this will be solved for other projects who want to migrate in the future at some point.

Migration day

The incredible support folks from OSUOSL helped us ensure that OAuth worked properly, among other things. And then the migration day came. Given that the test transfer took 18 hours, we started the migration at midnight. We gave a heads-up a few hours before on status.postmarketos.org, and shared that via IRC/matrix and Mastodon.

Then we did the following:

Timeout during merge request transfer

The import went the same as the test transfer … except the merge requests of pmaports, just this one repository, went into a timeout for several hundred merge requests. This was unfortunate, but at that point it would be unreasonable to start the transfer again and hope that everything gets imported this time. So now some of the merge requests are not fully transferred. However, the more important part is that all issues of all projects are imported.

It seems that the merge requests with timeout were not closed correctly, so now we had several hundred merge requests too many open. In addition to a lot of merge requests that should still be open, but were not usable for the most part since they were not owned by the users who originally submitted them any more. So they could not edit them and would need to submit new merge requests anyway, unless the authors happened to have push rights for the repository.

We decided to just close all open merge requests with a quick and dirty shell script:

#!/bin/sh
TOKEN="glpat-PUT-TOKEN-HERE"

while true; do
    curl \
        --header "PRIVATE-TOKEN: $TOKEN" \
        "https://yourgitlabinstance/api/v4/merge_requests?scope=all&state=opened" > open2.txt

    PROJECTS="$(cat open2.txt| jq  '.[].project_id' | sort -u)"
    echo "PROJECTS: $PROJECTS"

    if [ -z "$PROJECTS" ]; then
        echo "DONE!"
        exit 0
    fi

    for project in $PROJECTS; do
        OPEN_MRS="$(cat open2.txt| jq ".[] | select(.project_id == $project) | .iid")"
        echo "OPEN_MRS in $project: $OPEN_MRS"

        if [ -z "$OPEN_MRS" ]; then
            echo "DONE!"
            exit 0
        fi

        for i in $OPEN_MRS; do
            echo ":: $i"
            curl \
                --request PUT \
                --header "PRIVATE-TOKEN: $TOKEN" \
                "https://yourgitlabinstance/api/v4/projects/$project/merge_requests/$i?state_event=close"
            echo ""
        done
    done
done

Since then we manually re-opened a few merge requests that still are relevant and can be pushed to by the original authors (and merged most of them), but at least we don't have hundreds of irrelevant open MRs cluttering the instance.

Finishing up

Over the last week we fixed up several things, like CI not working and emails not arriving from our instance. Besides some smaller things everything seems to be working again. Let us know if something is not working for you in #77 or the postmarketos-devel chat.

Observations / lessons learned

Thanks to Oliver, Clayton, Luca, Stefan, Aster, for doing migration related tasks, and to Lance from OSUOSL for the amazing support, as well as everybody who has signed up to our new instance and started contributing to postmarketOS through it already!

If you appreciate the work we're doing with postmarketOS and want to support us, consider contributing financially via OpenCollective.