Consolidating Repositories

After the recent update to this server and a bit of retrospective on those services I both use and get value from I have been consolidating some old repositories to simply host them on this server rather than one of the numerous code forges. Here's a few notes on the mechanics of that.

A Target

I've mentioned that I've found fossil pretty painless to self-host so I figured I may as well consolidate to it, rather than pull in new technologies and tools. I've got a few old repositories in both git and mercurial, I'd like to maintain what little history exists in each repository. Mostly this is for my own memory as it may help correlate something I've previously written to the code itself (it should also be more representative of where I was at, chronologically, at the point of development).

Git to Fossil

Easy things first, fossil has a built-in import for a git export so dumping all the commits first into a single repository is pretty straightforward. Because none of the repositories were started with the idea that they'd need any kind of namespacing, most files in each repository exist at the root directory. In order to organize each repository at the time of import, I found git-filter-repo sufficient to move each project under a subdirectory so that the new project scopes each project under its own name in the file system hierarchy.

git -C "$tmp_clone" filter-repo --force --to-subdirectory-filter "$subdir"

Obviously $tmp_clone is a local clone I've made of the source repository. I wasn't convinced I wouldn't mess up the source repository in the migration so I cloned it locally before operating against it. The $subdir is just the name of the specific project I was namespacing.

Perhaps also obvious, the above command is operating in the context of a fresh git repository (that I called "workspace" while I was iterating on it). All of the git commits need to ultimately land somewhere and the goal is to put them into one git repo for reasons I'll explain in the next section.

Mercurial to Fossil

There's less of a straight line from Mercurial to Fossil. First I found I needed to translate Mercurial to Git, for which fast-export seemed to do a sufficient job. My usage was simple enough to be covered by the README but was just:

$ git init ${project}-git
$ cd ${project}-git
$ hg-fast-export.sh -r ../${project}
$ git checkout

From there the same git-filter-repo process covered the import into a single git repository.

Bundling It All Up

In practice I first converted the Mercurial repositories to git. Once I had several git repositories:

1. I used the filter-repo operation to place the repositories in a subdirectory
2. I added each repository as a remote to a new repository
3. I merged each remote (knowing that all the distinct files were namespaced under the filter-repo subdirecotires)
4. I removed each remote

WORK_TMP=$(mktemp -d)

for subdir in "${!PROJECTS[@]}"; do
  src="${PROJECTS[$subdir]}"
  tmp_clone="$WORK_TMP/$subdir"

  git clone "$src" "$tmp_clone"

  git -C "$tmp_clone" filter-repo --force --to-subdirectory-filter "$subdir"

  git remote add "$subdir" "$tmp_clone"
  git fetch "$subdir" --no-tags
  git merge --allow-unrelated-histories "$subdir/master" \
    -m "Merge $subdir into subdirectory $subdir/"
  git remote remove "$subdir"
done

Because I'm consolidating multiple independent repositories with no real shared history I had to find and use the --allow-unrelated-histories flag when merging:

--allow-unrelated-histories

By default, git merge command refuses to merge histories that do not share a common ancestor. This option can be used to override this safety when merging histories of two projects that started their lives independently. As that is a very rare occasion, no configuration variable to enable this by default exists or will be added.

Importing to Fossil

Finally, with one git repository of all the related histories it is a single step to import into a new fossil repository:

$ git fast-export --all | fossil import --git ../${new-fossil-repo}.fossil

Thoughts

This was passingly entertaining to explore a few different features of each technology. I am not certain how valuable it will really be long term. Partly as a result of the merge operations the repository "history" seems quite complicated in those visual interfaces I've looked at (the timeline view in the web UI and the fossil timeline output). I don't imagine that is terribly important as all of my own work is incredibly linear and unrelated to coincident changes. At least backups will be even easier now! Finally, the really good news is that the per-file history is easily navigable in my preferred VCS interface which is emacs vc-mode: