Common problems and their resolutions

This document is a work in progress! It should get filled in fairly fast though.

Platform deployment problems

Spring application deployment times out

Problem

A spring application times out during start with no indication of a problem. The last log message displayed is similar to the following:

2016-01-20T11:33:53.55+0000 [App/0] OUT 2016-01-20 11:33:53.554 INFO 29 — [ost-startStop-1] o.s.b.c.e.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/]

This happens more frequently on Openstack instances with a low number of compute nodes.

Resolution

Adding entropy to the hypervisor OS (i.e. the one running Openstack compute processes) solves the problem. While adding a hardware random number generator is preferable, the following solution also works:

sudo aptitude install rng-tools -y
sudo rngd -r /dev/urandom -o /dev/random

Please note that the above lowers the cryptographic strength of the keys generated by the application, so it is not recommended on production systems.

Platform operational problems

Error listing app instance numbers

Problem

When listing apps with cf a, app instance numbers show up as ?/1, for example:

user-management           started           ?/1         512M     1G     user-management.example.com
cdh-broker                started           ?/1         128M     1G     cdh-broker.example.com
hdfs-broker               started           ?/1         1G       1G     hdfs-broker.example.com
ipython-broker            started           ?/1         256M     1G     ipython-broker.example.com

Resolution

The resolution can be found here, under the Recovering from HM9000 Failure section. You can additionaly stop all hm9000 processes beforehand and start them in the following order: etcd1 -> hm1 -> etcd2 -> hm2.

Error parsing JSON

Problem

Can't list services in cf service-access when using the cf cli client:

Error parsing JSON: invalid syntax

Resolution

This is a bug in cf-cli 6.12.3. Downgrade do 6.12. Users of trustedanalytics/cloudfoundry-mkappstack need to set cfbinver in appstack.mk to 6.12.0

Cannot access manually spawned Centos instance on Openstack

Problem

When trying to SSH into an Centos instance that was manually created with the TAP provided image, you get:

Permission denied (publickey).

Resolution

You HAVE TO select the Configuration Drive option in the Advanced tab when creating an instance. This is used by the cloud-init scripts to get instance data such as authorized keys. More info available here: http://docs.openstack.org/user-guide/cli_config_drive.html

End-user problems

Error dialing loggregator

Problem

User can't tail logs of an application using cf logs app, gets the following error:

Error dialing loggregator server: Get https://loggregator.X.X.X.X.xip.io:443/recent?app=APPID: x509: certificate is valid for , not loggregator.X.X.X.X.xip.io.

Resolution

The api should be targeted with the --skip-ssl-validation flag, for example cf api api.X.X.X.X.xip.io --skip-ssl-validation.

Explanation

The root cause of this problem is an invalid or self-signed certificate for the domain the environment uses. This is common for testing instances using the xip.io domain.

Can't access dashboard for applications started via Marketplace

Problem

User can't access dashboard for applications started via Marketplace. HTTP code 500 is returned to browser.

Resolution

To fix this issue one need to:

1. Login to cdh-launcher instance
2. From cdh-lanucerh login to nginx-instance
3. On nginx-instance edit nginx.conf file via

sudo vim  /etc/nginx/nginx.conf
od
sudo nano /etc/nginx/nginx.conf

4. Change:

    proxy_buffering off;
    proxy_connect_timeout   180;
    proxy_send_timeout      180;
    proxy_read_timeout      900;

to:

    proxy_buffering off;
    proxy_connect_timeout   180;
    proxy_buffer_size        8k;
    proxy_send_timeout      180;
    proxy_read_timeout      900;

5. Restart nginx service via:

sudo service nginx restart