Re: Portmod - Mod Manager
Posted: 02 May 2019, 23:03
I’ve been working a lot on Portmod, and one of the few remaining features is patch support.
To recap, the idea was to use binary patches to handle in-place modifications to arbitrary files for purposes such as cleaning plugins, applying the modifications made by the TR-patcher and fixing bump maps on NIF files.
Unfortunately, looking further into the idea, binary patches really don’t seem to be a good tool for this. Unlike text patches, binary patches are extremely opaque, both due to the fact that the diff contained by the patch is binary, and the fact that none of the options I’ve seen (xdelta, bsdiff) support any sort of comments/annotations inside the files themselves.
We could go back to the idea of just having portmod run these scripts at runtime, possibly even including them in the repository, however we can’t include their dependencies in the repository (i.e. perl for tes3cmd, java for TR-patcher (admittedly common, though as usual I have idea as to the state of such dependencies on Windows), and pyffi for fix_nif (available via pip, but not through package managers)). I don't particularly like either the idea of including various tools in the mod repo, given that there are a lot of platform-specific issues with respect to installing them, or the idea of just having them be dependencies of Portmod, given that many mods won't need them and the dependency list just keeps growing. I think I have come up with a reasonable way of doing it, but I'm not entirely satisfied with it.
An alternative I’m considering, is creating patch files specifically formatted for plugin files. Basically a file containing a sequence of record/subrecord remove and add/modify commands, with text diffs for the contents of those records. Sort of like the diffs that esmtool can already produce, though without including the full details of removed records and having some nice wrapper information for descriptive purposes and to allow the patches to be self applying in the way that regular patches are. This wouldn't fix the bump map issue, but does reduce the number of special cases, especially given that pyffi and fix_nif are both implemented in python and would work nicely with a pip install of portmod.
The idea of plugin patches does bring us back to the issue of a native plugin library, previously discussed on the old thread with respect to omwllf. One thing that did occur to me is that we already have such a library within OpenMW itself, it’s just that OpenMW doesn’t currently provide the library and the relevant interfaces (or at least doesn't install it). I imagine that setting up such a library is probably not terribly high priority for the dev team at the moment, but it would be rather helpful. That being said, given that esplugin is already very close to providing all the necessary features, I’m really not sure whether writing a Rust interface for such a C++ “libopenmw” would be more or less work than just making the necessary changes to esplugin.
I think I’ll leave the issue of patches for a bit and start working on expanding the repository, as almost all other work on portmod 2.0 has been completed.
Also, thanks BaronPampa for the list of re-distributable mods. I'll probably add those to the repo first as I like mods with permissive licenses.
To recap, the idea was to use binary patches to handle in-place modifications to arbitrary files for purposes such as cleaning plugins, applying the modifications made by the TR-patcher and fixing bump maps on NIF files.
Unfortunately, looking further into the idea, binary patches really don’t seem to be a good tool for this. Unlike text patches, binary patches are extremely opaque, both due to the fact that the diff contained by the patch is binary, and the fact that none of the options I’ve seen (xdelta, bsdiff) support any sort of comments/annotations inside the files themselves.
We could go back to the idea of just having portmod run these scripts at runtime, possibly even including them in the repository, however we can’t include their dependencies in the repository (i.e. perl for tes3cmd, java for TR-patcher (admittedly common, though as usual I have idea as to the state of such dependencies on Windows), and pyffi for fix_nif (available via pip, but not through package managers)). I don't particularly like either the idea of including various tools in the mod repo, given that there are a lot of platform-specific issues with respect to installing them, or the idea of just having them be dependencies of Portmod, given that many mods won't need them and the dependency list just keeps growing. I think I have come up with a reasonable way of doing it, but I'm not entirely satisfied with it.
An alternative I’m considering, is creating patch files specifically formatted for plugin files. Basically a file containing a sequence of record/subrecord remove and add/modify commands, with text diffs for the contents of those records. Sort of like the diffs that esmtool can already produce, though without including the full details of removed records and having some nice wrapper information for descriptive purposes and to allow the patches to be self applying in the way that regular patches are. This wouldn't fix the bump map issue, but does reduce the number of special cases, especially given that pyffi and fix_nif are both implemented in python and would work nicely with a pip install of portmod.
The idea of plugin patches does bring us back to the issue of a native plugin library, previously discussed on the old thread with respect to omwllf. One thing that did occur to me is that we already have such a library within OpenMW itself, it’s just that OpenMW doesn’t currently provide the library and the relevant interfaces (or at least doesn't install it). I imagine that setting up such a library is probably not terribly high priority for the dev team at the moment, but it would be rather helpful. That being said, given that esplugin is already very close to providing all the necessary features, I’m really not sure whether writing a Rust interface for such a C++ “libopenmw” would be more or less work than just making the necessary changes to esplugin.
I think I’ll leave the issue of patches for a bit and start working on expanding the repository, as almost all other work on portmod 2.0 has been completed.
Also, thanks BaronPampa for the list of re-distributable mods. I'll probably add those to the repo first as I like mods with permissive licenses.