The formula used for the centroid of a spherical triangle in
gmtlib_geo_centroid_area (take the 3d mean of the three vertices and
normalize). An example where you clearly get the wrong answer is given
by whuber in
https://gis.stackexchange.com/questions/43505/calculating-a-spherical-polygon-centroid
The right formula is given by Brock (1975)
https://doi.org/10.1115/1.3423535
This easily generalizes to a polygon (without the need to break the
polygon into triangles).
While on the subject of gmtlib_geo_centroid_area, I'll note that the
method of computing the area (a direct calculation of the spherical
excess) is terribly inaccurate for small areas. I recommend instead the
formula given in
https://en.wikipedia.org/wiki/Spherical_trigonometry#From_latitude_and_longitude
Here again this formula can naturally be applied to a polygon (although
care needs to be taken if the polygon encircles a pole).