Change import path to github.com/ugorji/go-codec about go

[ugorji]

Keeping the repo name consistent with the package path has resolved a spurious issue arising in #289 (see #289 (comment) ).

This is a good reason to leave well enough alone.

@dmitshur ^^

[dmitshur]

This sounds like a good change.

We hope to do this soon, likely by the end of january.

Any updates?

[ugorji]

Sorry, no updates yet.

I'm not yet convinced that the benefits outweigh the compatibility cost.

[warpfork]

+1 to rename, fwiw.

[drewwells]

Also, forking your project becomes less practical, since it requires github.com/{myname}/go is available.

[ebehar]

Adding on to other comments above from last year, remedying the compatibility change is quite easy, and renaming the repository would be much cleaner for consumers of it.

I feel like everybody should be able to search and replace using their favorite tools to change the import paths pretty easily.

If that's concerning though, could you alternately fork the repo to go-codec, move things around to make it the way you want it, and then make go-codec the canonical repo for these codecs? That way consumers of the package can migrate to the new repo name on their own time, and the old repo can be deleted eventually.

[ugorji]

Currently, per sourcegraph, over 400 repositories depend on this library. Making such a change will be extremely disruptive to folks.

Unfortunately, I have thought long and hard about this, and there's no clean way to do it without un-necessary disruption. I liken this to backward-compatibility: sometimes it sucks, but it is necessary to carry forward.

Closing for now.

[arizvisa]

The following was referenced in #275.

This was left unanswered in the prior closed issue, but is there some reason why you don't just keep the original repository name, and just add another repo as go-codec?

You can setup git to push to two different remotes at the same time so it really doesn't cost you anything (such as having to double-maintain), and since the repository name is optional (a user can choose to use the suggested name or the prior backwards-compat name), it really doesn't affect users/stale-projects and hence doesn't affect backwards compatibility during building in any way.

But this way at least people have an option to migrate to the new name over time. A number of your forks exist explicitly for renaming it to "go-codec" or "go-msgpack", so your users have definitely recognized that your initial project name "go" was a mistake. You could even point people to the https://gopkg.in url of your project in your docs to include the version as part of the package name.

This library is pretty much yours so you can do whatever you want with it of course, but there will definitely come a time when you will stop maintaining this project. And if this project follows the traditional open-source lifecycle, then someone else will likely inherit it. If somebody else implements a project called "go-codec" then they also have the option to impersonate your project. Before that happens, however, it'd be nice to get a solution for that the namespace collision available before that point happens I guess.

Anyways, it's all up to you of course. I just saw the comments/discussion that happened and don't think that aliasing the name of a project written in a relatively young language such as go is as intrusive as one might believe.

(edited to remove some unnecessary verbiage)

[ugorji]

In retrospect, I am sort of happy that this change wasn't done, for the following reasons:

• Single (mono) repo for storing all go libraries/tools, analogous to github.com/golang/tools
• Compatibility: every user has to change their code
• Source code churn: changing the package name involves moving everything one directory up, which completely messes up the git history

Let me hit on each one by one

Single Repo

With modules, it is possible now to have multiple packages inside the same repository.

Also, we now have some precedent with the https://github.com/golang/tools, which is a single repository that hosts the source for multiple tools managed by the core go team, including blog, cover, godoc, imports, present, playground, etc. Module support will help version the different tools, even though they all are hosted in a single github repo.

This also aligns with the way I hear code is stored in google and facebook, and the react library, in a single mono-repo.

The analogy is that my go code that I share with the world is stored at github.com/ugorji/go, and includes the codec library, codecgen tool, etc.

That's one part.

Compatibility

If we change the package name, then every user has to update their code, or we split the userbase into old and new. We see how that has not worked out well when folks do it, and my package is not as popular that everyone will happily bite the bullet.

Look at https://godoc.org/github.com/ugorji/go/codec (scroll to the bottom): it says that my package is now used by 2214 packages. That is way too much to force a change to. And we still have no defined way to make announcements.

Source code churn

Changing the package name involves moving everything one directory up, which completely messes up the git history. I don't want to do it unless I have to.

For these reasons, making the change is a no. Folks are generally ok with it, and the ecosystem (with modules, etc) now encourages this model of a single repo containing multiple packages - unlike before where this wasn't truly supported.

Hope this helps explain the position.

In the future, I hope a service comes that provides true canonical package names in a flexible way. Currently, the "import paths metadata" doesn't support this, as it still expects that the canonical name and the repo have the same base path. Consequently, I can use it to map domainName/go/codec to github.com/ugorji/go/codec, but not to map domainName/go-codec. I would hope that the "godoc.org" service provides something like this, but it requires more flexible support from the go tool first.

I may follow up on this with the go tools teams in the future.

[ugorji]

@arizvisa ^^ (see comment above)

[arizvisa]

Fair enough. Extending on your points though:

• Single-Repo
  The difference is that golang/tools is a number of tools/commands with shared code, whereas your go repo is (at the moment) for a single library. This'd be consistent if your other go project (go-codec-benchmark) would be in this same repo, but you keep them as separate distinct projects. You can retain git history and leave each reference as a submodule so that everything is accessible from the same place. But again, it's just something to think about as you develop more successful libraries in the future.

• Compatibility
  It's not actually a change, but rather an alias. This is just due to your naming of the project as go-codec which is just different from everything that references it themselves. But you made your point. It is just aesthetics after all. ;)

• Source code churn
  You shouldn't need to move everything up which'll force users to change code. You can even keep it as a submodule (an alias to the actual project) and in no way require users to shuffle code around. To ease your development you can push to both places at the same time. (Commit history can also be retained using git filter-branch when moving it into a separate github project.) But, this way the git history for each different go project can still be tracked and everything under user/go could be what you'd consider "stable" (in a sense).

Tbh, I prefer your method of using user/go to organize all go code libraries underneath it. As this way the separation of development/experimental and stable branches can be controlled by synchronizing a submodule of "stable" code under one's user/go repository. This can be used to deal with versioning in a project repo and since go get doesn't recognize tags/branches. This solves the problem of go get always just fetching the latest commit and simply ignoring what we tag in git. Of course, though, the work for this only makes sense when there's more than a 1-man army working on a project.

As an example, like even tho you want to complete deprecate go-msgpack (and you've made a lot of effort to deprecate that library). To retain backwards compat (even tho I think your effort has been successful), you could keep a submodule in user/go as user/go/msgpack which references go-msgpack, and another one user/go/codec that references go-codec. This way you have no fears with users who're still using go-msgpack.

Anyways, thanks for your work on go-codec, it's a great little library.

(edited to mention go-msgpack and to format bulletpoints)

[ugorji]

@arizvisa

Honestly, my biggest concerns are source code churn and compatibility.

Compatibility

If I change the repo name, then I am suggesting that folks change the import path in their codebase. This effectively splits the ecosystem, which go doesn't allow. I cannot have the same codebase support github.com/ugorji/go-codec/codec and github.com/ugorji/go/codec . This is even more critical with the advent of modules, where you MUST specify the module name in the go.mod file. This is why the "import path" comment was created since go 1.4, so that you could have an "alias" but enforce a single import path.

Source code churn

If I change the import path, then I WOULD like to remove the codec subdirectory, and move everything one directory up. That means that "github.com/ugorji/go/codec" would move to "github.com/ugorji/go-codec" with package name "codec". This would cause unnecessary source code churn, and cannot be solved while solving compatibility above i.e. it goes against compatibility above.

[ugorji]

I will open this issue back up, because it will be nice to solve it somehow. But if no clear solution is found, then I will just leave it as a reminder and let it be the legacy issue that cannot be solved.

[ugorji]

I am currently considering a middle ground - but am not sold on doing it yet.

1. Change repo name to go-codec
2. github will transparently redirect go to go-codec
3. Add an import path so that it MUST be imported as github.com/ugorji/go/codec
4. DONE

This way, we change the repo name to match the function of the repo. Yet we maintain compatibility and have no source code churn.

I do not want to use submodules because they bring inconvenience, but seen now and unknown. Some of these concerns are nicely articulated at the bottom of https://github.blog/2016-02-01-working-with-submodules/

[ugorji]

Updated by 74d13ae

[dcarbone]

This needs to be addressed asap. You pushed a breaking change in early march (moving the location of RawToString breaks packages that tried to use literal construction i.e. &codec.MsgpackHandle{RawToString: true}) that is causing me endless headaches. I have a dependency that uses this package as a dependency and while I have an accepted pr that addresses this (uses new() -> set syntax instead), it has not been merged into a stable tag.

This is a VERY big issue now that you have released another tag with this breaking change. Because you define your import path as github.com/ugorji/go, but do not have any actual code there, neither dep nor mod will allow me to constrain this repo properly.

edit dep will, thanks to the [[override]] block, but mod just produced headaches. I really hope something can be done here.

I am sorry if I sound harsh, but I have been fighting this for a month now and I grow weary of it.

[ugorji]

@dcarbone While I can understand the tendency to lash out when frustrated, I cannot accept this here. Next time you need to sound harsh and lash out, close a door and scream. Then compose yourself and make your point. It will go down much nicer next time.

This is the way this works, and anyone that has submitted a bug report can attest to this. You find an issue, you describe it, I resolve it typically within a few days. There's a reason why the bug count is typically about 0 (save for these reminder bugs I keep around).

Do us all a favor. Next time something bothers you, articulate your issue swiftly - and watch it get resolved very quickly. If you had brought the issue to my attention earlier, we might have resolved it before it stayed for so long. The main issues here is that I forgot that folks would use the literal constructors, and so that code is broken. That is a big shame - I wish someone had raised the flag early since I didn't realize it.

About the other thing i.e. dep and mod having an issue with the module path of github.com/ugorji/go - articulate your issue well in a new bug and I can try to address it. It may be as simple as putting a holder file in the module path.

But I can't afford this ridiculousness you did above. Behavior like that serves only as a detriment to the entire community, as it discourages folks from doing good work for free.

Hope this message reaches you well.

[dcarbone]

I believe I articulated my issues very concisely. Using go mod, it is not possible to constrain this package properly as there is no code present in the path defined by the mod file. In addition, you cannot merely use a typical [[constrain]] block in a Gopkg.toml because, again, there is no code in the root of the repo. The same goes for Glide. All of these issue would be fixed if you were to change the package path as defined in this issue.

If you really want, I will open separate issues.

[ugorji]

I am currently considering a middle ground - but am not sold on doing it yet.

1. Change repo name to go-codec
2. github will transparently redirect go to go-codec
3. Add an import path so that it MUST be imported as github.com/ugorji/go/codec
4. DONE

This way, we change the repo name to match the function of the repo. Yet we maintain compatibility and have no source code churn.

I do not want to use submodules because they bring inconvenience, but seen now and unknown. Some of these concerns are nicely articulated at the bottom of https://github.blog/2016-02-01-working-with-submodules/

Many tools e.g. travis-ci, etc do not work well with a repo name different from the import path. It prevents us from getting this testing with multiple versions of go officially. Due to this, I will revert the repo name back to go.

[dmitshur]

travis-ci, etc do not work well with a repo name different from the import path

Travis CI supports this well via the go_import_path field. See https://docs.travis-ci.com/user/languages/go/#go-import-path.

Look at .travis.yml of github.com/go4org/go4 for an example of a repository with a vanity import path using it.

[trivigy]

@ugorji one really simple way to solve this whole issue and avoid the separate repo is to tag the sub-directory with it's own versioning. If you do tags: codec/v1.1.1, codec/v1.1.4, etc then when people do go get -u github.com/ugorji/go/codec or go get -u github.com/ugorji/go/codec@latest instead of getting auto-generated commit based tag, it will just fetch the actual latest tag.

Note that this issue is happening across a lot of repositories that are using this module. This is primarily happens because people are committing indirect references in their go.mod.

Super simple fix and doesn't require any file or repo relocation at all. Which, I agree with you that it shouldn't be done, will cause a whole lot more breakage.

[ugorji]

Wow. I didn't know about this. Can you send me some documentation about it? Or link to source? This is awesome if this works. Thanks so much.

[trivigy]

@ugorji https://github.com/golang/go/wiki/Modules#publishing-a-release Read the last paragraph.

For example, suppose we have a module example.com/repo/sub/v2, and we want to publish version v2.1.6. The repository root corresponds to example.com/repo, and the module is defined in sub/v2/go.mod within the repository. The prefix for this module is sub/. The full tag for this release should be sub/v2.1.6.

I use it with one of my libraries. https://github.com/syncaide/flats I have sub-sub-sub folder for my actual module because of some nonsense auto-generation stuff. Basically I have a src folder that holds template files which are then processed and generated into golang and typescript located in dist folder, which I ultimate commit. In order to import the library from within my other projects (this is a supporting library) I have to do something like go get -u github.com/syncaide/flats/dist/go/flats.

p.s. also you will need to add a go.mod in the codec folder.

Before the tagging it used to produce something like:

module github.com/syncaide/gate

require (
	github.com/syncaide/flats/dist/go/flats v0.0.0-20190513213919-e649cce56cd2
)

After tagging it started doing:

module github.com/syncaide/gate

require (
	github.com/syncaide/flats/dist/go/flats v0.2.11
)

[ugorji]

Wow. I didn't know about this. Can you send me some documentation about it? Or link to source? This is awesome if this works. Thanks so much.

[dmitshur]

@ugorji That https://golang.org/wiki/Modules wiki page has lots of documentation about multi-module repositories. There's also a specific FAQ section at https://golang.org/wiki/Modules#faqs--multi-module-repositories.

[ugorji]

@dmitshur I will open up an issue specifically about the use of modules.

It has been a disaster, and I really wish I had waited until the dust settled before making this module aware, especially since so many people depended on this package and it has the atypical "package name differs from directory name" issue.

I hope folks can comment there directly. I will reference your messages.

[ugorji]

Hi folks,

Please see #299 and add your thoughts/ideas/etc. Thanks.

[ugorji]

FYI: I just released a go-codec production release - version 1.1.7 (finally)

First, it resolves the go.mod impasse where we had different import paths (github.com/ugorji/go and github.com/ugorji/go/codec) causing the ambiguous import error.

This is now fixed by leveraging import cycles to ensure that either one works well and resolves to the same bits.

Please let me know if seeing any issues. If all is well over the next few days, I will close this github issue.

• https://github.com/ugorji/go/releases/latest
• https://github.com/ugorji/go/releases/tag/v1.1.7

[ugorji]

Closing this issue as the modules impasse has been resolved.

Thanks for all your assistance.