Issues accessing Fastly endpoints from within docker that runs in Gitpod

[shaal]

Bug description

Issues accessing Fastly endpoints from within docker that runs in Gitpod

When running curl https://www.drupal.org/files/issues/2020-03-30/2759397-25.patch -vvv, I am getting -

*   Trying 151.101.2.217:443...
* Connected to www.drupal.org (151.101.2.217) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

And then it hangs for a minute, and display

* OpenSSL SSL_connect: Connection reset by peer in connection to www.drupal.org:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to www.drupal.org:443

Steps to reproduce

• Open any Gitpod workspace (tested in US Gitpod workspace)
• Confirm 10 times that you can run curl https://www.fastly.com with no issues ✅
• docker run -it debian:11
• apt update
• apt install curl -y
• Confirm 10 times that you can run curl https://www.fastly.com with no issues

You'll notice that inside docker, sometimes it works perfect, and sometimes it just hang up

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

(This was found because of new errors when running composer requests to drupal.org (which uses Fastly CDN) through DrupalPod.

This might be a temporary Fastly issue? but it's weird why I cannot replicate this in Gitpod itself (not inside docker inside Gitpod), nor I can replicate that on local machine with or without Docker.

[shaal]

@misterfalcon posted this on Gitpod Discord, not sure if it's related or not -
https://discord.com/channels/816244985187008514/839379835662368768/964597530648252476

have a question that I also sent to support, but maybe someone here has some insight....
Outbound connections to google over https are not working all of a sudden (starting today) from within our docker
containers. These same docker containers are also running on EC2 instances that we manage and have setup through a
CI/CD pipeline, so effectively the same setup. We are not having any issues on our EC2 instances so we suspect there's
something going on with Gitpod. Yesterday we had no issues, and today we did with no change to our code or config.

These are strictly issues reaching google, which affects our API since that relies on making external calls to Firebase. All
outbound calls to google from our container are hanging on TLS handshake OUT. We tested this by calling curl -v
https://www.google.com/ from within our container.

Any ideas?

[motiapp]

We think this is a Docker config issue on Gitpod's end relating to DNS resolution. Running cat /etc/resolv.conf in the container returns:
nameserver 127.0.0.11
options ndots:0

When running on our EC2 instance we get:
search ec2.internal
nameserver 127.0.0.11
options edns0 trust-ad ndots:0

From what I've read Docker is supposed to mirror /etc/resolve.conf from the host machine to the container when the container starts up. The host machine in a gitpod workspace reports a good config:
nameserver 1.1.1.1
nameserver 8.8.8.8

[aledbf]

@motiapp what container are you using to get that output (I cannot reproduce the error)

[motiapp]

@motiapp what container are you using to get that output (I cannot reproduce the error)

These are containers we setup in our repo and that get prebuilt and run when the workspace starts up.

[shaal]

This problem was partially fixed by #9356, now it doesn't happen in Docker.
But project that use docker-compose, still have the same issue.

It wasn't like that before today.

[motiapp]

We also are using docker-compose, and still seeing the issue. What makes no sense to me is I can get good responses from all sorts of sites using curl inside the container, except google.

[motiapp]

Here's a repo you can use to prove the failure I'm seeing:
https://github.com/motiapp/docker-test

I also added a branch that starts the container user docker run instead of docker-compose up to show that the issue I'm seeing is only occurring when using docker-compose up.
https://github.com/motiapp/docker-test/tree/fix

[shaal]

According to @aledbf, this is a side effect of removing slirp4netns #8955 (comment)

A fix for docker was implemented yesterday, #9356

But it doesn't solve the problem for projects that use docker-compose, because docker-compose does not follow the value from dockerd.

A solution/workaround for docker-compose is being worked on here -
#9360

In the meantime, on the project level it can be solved by adding this docker-compose yaml file:

networks:                                
  default:                               
    driver_opts:                         
      com.docker.network.driver.mtu: 1440

[motiapp]

Excellent! I can confirm the network config works for me.

[aledbf]

Closing. Fixed by #9360 and #9356

@motiapp please open a workspace and run docker-compose version
Should return Docker Compose version v2.4.1.2-gitpod.

No change is required to the docker-compose yaml file.