Repositories should be identified and retrieved via URL rather than name

[kriswest]

Describe the bug
This issue relates to/overlaps with two past unresolved issues:

• Provide facility to manage multiple forks of the same upstream repository #66
• Git Proxy does not support GitLab repos #511
  • fix: add gitlab support #545

Throughout the codebase repositories are assumed to have a unique name property. However, repository names are not globally unique and there are two common cases where they will be duplicated:

1. Multiple forks of the same project (e.g. within Github, bearing two different user's names as the 'organisation' (issue Provide facility to manage multiple forks of the same upstream repository #66)
2. Different projects that have the same name, hosted in different platforms.

• This is part of, but not the whole of issues fix: add gitlab support #545 - the same change is needed.

What we can rely on being unique is the repository URL being pushed to.

In PR #545 where @msagi was looking at this, it is suggested that we move to a platform/org/reponame structure. However, this requires additional processing of the URL and assumes that the platform/org can be extracted from the URL - the org is a GitHub/GitLab concept which doesn't apply to other hosts/base git repos.

On the other hand, a URL is globally unique and already exists within the data model. Switching to it is to identify repositories is easy enough - except where it impacts the URLs used to contribute to those repos through git proxy, e.g.:
git clone http://localhost:8000/finos/git-proxy.git
would need to become:
git clone http://localhost:8000/https://github.com/finos/git-proxy.git

This URL is uglier but easier to understand - the git proxy url has simply been prepended to it.

The challenge would be that you can't automatically migrate existing users with a checkout configured with the eexisting URL as origin (at least AFAIK - github manages repo URL migrations so there may be a way in the git protocol - although I suspect they just maintain some redirects). Hence, to solve this without a breaking change, there may need to be some fallback code to handle older checkouts and/or some form of migration. I'd like to figure out the preferred route forward on that before undertaking work to resolve (as I can see two others took this on and didn't bring it to ground). Options:

1. Fallback to existing approach: Switch to repo URL for retrieval and if repo not found/URL didn't contain the full url (so no 'http' in the string) attempt lookup by name.
2. Advise users to migrate: Create a table of the existing repo names at the time of migration. Switch to repo URL for retrieval and if repo not found, check the table and respond with a message advising user to migrate to the new URL.
3. Issue a breaking change and advise firms updating to inform all users that their repos need to migrated to the new origins.

Expected behaviour

• Multiple forks of a project can be onboarded, managed and contributed to separately
• Multiple repos with the same name, but oowned by different orgs or hosted on different platforms can be onboarded, managed and contributed to separately.

Additional context
I may be up for implementing a solution to this issue - perhaps building @msagi's previous attempt in #545. However, i want to agree the approach before taking it on.

We ran across this in initial testing with devs using forks of the same repo for testing. It will come up less often in production use - but it will come up again someday... As the solution is likely to be disruptive or breaking I'd suggest this is sorted sooner rather than later when it will affect more organisations and users... and potentially will need to be address in more places as git proxy itself grows.

[kriswest]

Caught up with a few people from Gitlab's co-create team that were aware of GitProxy not supporting GitLab repos. Perhaps we can encourage them to contribute to a fix along these lines. I'd be interested on opinions about making this change @JamieSlome @grovesy @coopernetes (before I consider trying to implement it)

[StingRayZA]

I'd be in support of adding the 'full' URL behind the proxy url as suggested (e.g. http://git-proxy:8000/github.com/finos/git-proxy.git) While gitlab.com is certainly a desired target, this would enable contributions to other gitlab instances (e.g. https://salsa.debian.org, https://git.drupalcode.org amongst many others) and as well as code hosted on other (git) platforms.

Additionally, code on GitLab instances is not organised simply with an 'organisation' / 'project' layout, but could have (multiple) levels of subgroup in between, for example https://gitlab.com/gitlab-org/ruby/gems/gitlab-triage would have a proxy url of http://git-proxy:8000/gitlab.com/gitlab-org/ruby/gems/gitlab-triage.

This could be introduced in a backwardly compatible fashion, i.e. a url of http://git-proxy:8000/finos/git-proxy.git would 'fall back' to github.com. That said - any instances where a fork of the same project on github.com was added to git-proxy before this change is implemented, would still be 'collapsed' into one. After this change was deployed, those 'new' forks would be fully supported.

[coopernetes]

My stance is:

1. Explicit intent encoded in URLs should be preferred as it reduces the ambiguity as to what upstream git repository server the proxy is handling. I fully support inclusion of the hostname in the proxied routes.
2. Avoid special characters in URLs - this likely isn't a problem for vast majority of git clients but it's an edge case I'd rather not have to deal with on the client side.
3. Until we integrate expanded protocol support (such as what's proposed via feat: adds ssh support for git operations #987), HTTP/S is the main transport of choice since we have the framework (Express) and middlewares needed without too much hand-rolled layers in GitProxy. With the introduction of additional protocols, that is guaranteed to be a separate server (at the very least) if not an entirely separate running process under the main GitProxy "app". In that event, the URLs can be colliding since it's separate TCP ports that the server is listening on and they are independent of one another.
4. Internally, we can be hyper explicit by storing the protocol as a field in the database and default it to https.
5. There may be open source projects hosted on plain HTTP which we don't want to cutoff exactly or have zero support for proxying. It can be disabled by default and opted in via configuration. In those instances, we store the protocol as http in the backend so its clear that this is an insecure upstream. These days, its trivial enough to setup TLS via a provider like LetsEncrypt so all upstream git servers should be using TLS already. Having this distinction both from a GitProxy git server configuration as well as upstream proxying support is useful even with the current HTTP-only based implementation.

For backwards compatibility, we can inspect the incoming URL and determine if the first URL part contains a hostname. This is a trivial enough check using a simple library like validator:

// within a request parsing function
const firstPart = req.url.split("/")[1]

console.log(firstParth);
// either:
//  "github.com" or
//  "{owner}"
const validator = require('validator');

// validator.isFQDN(string) requires the string to be fully qualified, e.g. contains a dot "." character. GitHub repo URLs do not allow for dots and other special characters
console.log(validator.isFQDN(firstPart)); // if true, parse owners and names by first stripping the leading "/{upstreamHostname}"

// current behaviour
console.log(validator.isFQDN('finos')); // false

I can also imagine that certain features such as whitelisting (authorisedList) should be "owner" or "org" aware. I've experimented with this in a Java-based implementation. Ignore the language for now - the abstraction proposed is what I'd like feedback on. It would be incredibly useful to decouple this setup with specific SCM hosting providers while still leveraging some of the provider specific behaviour for filtering features. It is desirable to whitelist based on these contextually significant parts of an incoming/outgoing request URL. See below example:

public interface GitProxyProvider {
    String getName();
    URI getUri();
    default String getPath(URI upstreamUri) {
        return "/" + upstreamUri.getHost();
    }
}
...
public inteface UrlBasedOwnerProvider {
    // requestUri: /github.com/foo/bar.git
    default String getOwner(URI requestUri) {
       return uri.split("/")[2]; // returns "foo"
    }
}
...
public interface UrlBasedNameProvider {
    // requestUri: /github.com/foo/bar.git
   default getRepository(URI requestUri) { // name of the repository if it shows up in the URL at all
       return uri.split("/")[3].replace(".git", ""); // returns "bar"
   }
}
...
public class GitHubProvider implements UrlBasedOwnerProvider, UrlBasedNameProvider {

    public GitHubProvider() {
        this.name = "github";
        this.uri = "https://github.com";

    }

    // Custom GitHub installations that still use "/{hostname}" as the server path
    public GitHubProvider(String name, URI uri) {
        this.name = name;
        // ..
    }

    // use the default methods of interfaces
}
...
public class GenericProvider implements GitProxyProvider {
   // doesn't implement any URL significant logic...
}
...
// This class implements whichever interface applies based on how the server is deployed and what URL(s) a git repo is hosted on
public class ExampleBespokeProvider implements UrlBasedOwnerProvider {
   
    public ExampleBespokeProvider() {
        this.name = "bespoke";
        this.uri = "https://bespoke-hosted-git-server.example.com/foo/bar/baz/qux";
    }
    // ...
    @Override
    public String getOwner(URI requestUri) {
        // implement whatever custom logic that applies for this provider...
    } 
}

You get the idea. We will need these sorts of abstractions to allow individual deployments of GitProxy to be adapted to a wide variety of use cases.