Next release version (#1)

* add next release version increase udf class.

* delete fastutil dependency, directly extract intarrays code into project, reduce generate jar file size.

* add some bitwise functions

* add some map functions

* add day_of_year, geo, url functions and some unit test.

Author: aaronshan

README-geo.md

@@ -0,0 +1,27 @@
+## 当前互联网地图的坐标系现状
+### 地球坐标 (WGS84)
+- 国际标准，从 GPS 设备中取出的数据的坐标系
+- 国际地图提供商使用的坐标系
+
+### 火星坐标 (GCJ-02), 也叫国测局坐标系
+- 中国标准，从国行移动设备中定位获取的坐标数据使用这个坐标系
+- 国家规定： 国内出版的各种地图系统（包括电子形式），必须至少采用GCJ-02对地理位置进行首次加密。
+
+###百度坐标 (BD-09)
+- 百度标准，百度 SDK，百度地图，Geocoding 使用
+- (本来就乱了，百度又在火星坐标上来个二次加密)
+
+## 开发过程需要注意的事
+- 从设备获取经纬度（GPS）坐标
+    * 如果使用的是百度sdk那么可以获得百度坐标（bd09）或者火星坐标（GCJ02),默认是bd09
+    * 如果使用的是ios的原生定位库，那么获得的坐标是WGS84
+    * 如果使用的是高德sdk,那么获取的坐标是GCJ02
+- 互联网在线地图使用的坐标系
+    * 火星坐标系：
+        + iOS 地图（其实是高德）
+    	+ Google 地图
+    	+ 搜搜、阿里云、高德地图
+	* 百度坐标系：
+    	+ 当然只有百度地图
+	* WGS84坐标系：
+        + 国际标准，谷歌国外地图、osm地图等国外的地图一般都是这个

README.md

@@ -35,7 +35,7 @@ It will generate hive-third-functions-${version}-shaded.jar in target directory.
 
 You can also directly download file from [release page](https://github.com/aaronshan/hive-third-functions/releases).
 
-> current latest version is `2.1.0`
+> current latest version is `2.2.0`
 
 ## Functions
 
@@ -51,31 +51,41 @@ You can also directly download file from [release page](https://github.com/aaron
 
 | function| description |
 |:--|:--|
-|array_contains(array<E>, E) -> boolean | whether array contains value or not.|
+|array_contains(array&lt;E&gt;, E) -> boolean | whether array contains value or not.|
 |array_intersect(array, array) -> array | returns the two array's intersection, without duplicates.|
-|array_max(array<E>) -> E | returns the maximum value of input array.|
-|array_min(array<E>) -> E | returns the minimum value of input array.|
+|array_max(array&lt;E&gt;) -> E | returns the maximum value of input array.|
+|array_min(array&lt;E&gt;) -> E | returns the minimum value of input array.|
 |array_join(array, delimiter, null_replacement) -> string | concatenates the elements of the given array using the delimiter and an optional `null_replacement` to replace nulls.|
 |array_distinct(array) -> array | remove duplicate values from the array.|
-|array_position(array<E>, E) -> long | returns the position of the first occurrence of the element in array (or 0 if not found).|
-|array_remove(array<E>, E) -> array | remove all elements that equal element from array.|
+|array_position(array&lt;E&gt;, E) -> long | returns the position of the first occurrence of the element in array (or 0 if not found).|
+|array_remove(array&lt;E&gt;, E) -> array | remove all elements that equal element from array.|
 |array_reverse(array) -> array | reverse the array element.|
 |array_sort(array) -> array | sorts and returns the array. The elements of array must be orderable.|
 |array_concat(array, array) -> array | concatenates two arrays.|
-|array_value_count(array<E>, E) -> long | count array's element number that element value equals given value.|
+|array_value_count(array&lt;E&gt;, E) -> long | count array's element number that element value equals given value.|
 |array_slice(array, start, length) -> array | subsets array starting from index start (or starting from the end if start is negative) with a length of length.|
-|array_element_at(array<E>, index) -> E | returns element of array at given index. If index < 0, element_at accesses elements from the last to the first.|
+|array_element_at(array&lt;E&gt;, index) -> E | returns element of array at given index. If index < 0, element_at accesses elements from the last to the first.|
 
-### 3. date functions
+### 3. map functions
+| function| description |
+|:--|:--|
+|map_build(x&lt;K&gt;, y&lt;V&gt;) -> map&lt;K, V&gt;| returns a map created using the given key/value arrays.|
+|map_concat(x&lt;K, V&gt;, y&lt;K, V&gt;) -> map&lt;K,V&gt; | returns the union of two maps. If a key is found in both `x` and `y`, that key’s value in the resulting map comes from `y`.| 
+|map_element_at(map&lt;K, V&gt;, key) -> V | returns value for given `key`, or `NULL` if the key is not contained in the map.|
+|map_equals(x&lt;K, V&gt;, y&lt;K, V&gt;) -> boolean |  whether map x equals with map y or not.|
+
+### 4. date functions
 
 | function| description |
 |:--|:--|
 |day_of_week(date_string \| date) -> int | day of week,if monday,return 1, sunday return 7, error return null.|
+|day_of_year(date_string \| date) -> int | day of year. The value ranges from 1 to 366.|
 |zodiac_en(date_string \| date) -> string | convert date to zodiac|
 |zodiac_cn(date_string \| date) -> string | convert date to zodiac chinese | 
 |type_of_day(date_string \| date) -> string | for chinese. 获取日期的类型(1: 法定节假日, 2: 正常周末, 3: 正常工作日 4:攒假的工作日),错误返回-1. |
 
-### 4. json functions
+### 5. json functions
+
 | function| description |
 |:--|:--|
 |json_array_get(json, jsonPath) -> array(varchar) |returns the element at the specified index into the `json_array`. The index is zero-based.|
@@ -86,7 +96,17 @@ You can also directly download file from [release page](https://github.com/aaron
 |json_extract_scalar(json, jsonPath) -> array(varchar) |like `json_extract`, but returns the result value as a string (as opposed to being encoded as JSON).|
 |json_size(json, jsonPath) -> array(varchar) |like `json_extract`, but returns the size of the value. For objects or arrays, the size is the number of members, and the size of a scalar value is zero.|
 
-### 5. china id card functions
+### 6. bitwise functions
+
+| function| description |
+|:--|:--|
+|bit_count(x, bits) -> bigint | count the number of bits set in `x` (treated as bits-bit signed integer) in 2’s complement representation |
+|bitwise_and(x, y) -> bigint | returns the bitwise AND of `x` and `y` in 2’s complement arithmetic.|
+|bitwise_not(x) -> bigint | returns the bitwise NOT of `x` in 2’s complement arithmetic. | 
+|bitwise_or(x, y) -> bigint | returns the bitwise OR of `x` and `y` in 2’s complement arithmetic.|
+|bitwise_xor(x, y) -> bigint | returns the bitwise XOR of `x` and `y` in 2’s complement arithmetic. | 
+
+### 7. china id card functions
 
 | function| description |
 |:--|:--|
@@ -98,6 +118,27 @@ You can also directly download file from [release page](https://github.com/aaron
 |is_valid_id_card(string) -> boolean |determine is valid china id card No.|
 |id_card_info(string) -> json |get china id card info. include province, city, area etc.|
 
+### 8. geographic functions 
+
+| function| description |
+|:--|:--|
+|wgs_distance(double lat1, double lng1, double lat2, double lng2) -> double | calculate WGS84 coordinate distance, in meters. |
+|gcj_to_bd(double,double) -> json | GCJ-02(火星坐标系) convert to BD-09(百度坐标系), 谷歌、高德——>百度|
+|bd_to_gcj(double,double) -> json | BD-09(百度坐标系) convert to GCJ-02(火星坐标系), 百度——>谷歌、高德|
+|wgs_to_gcj(double,double) -> json | WGS84(地球坐标系) convert to GCJ02(火星坐标系)|
+|gcj_to_wgs(double,double) -> json | GCJ02(火星坐标系) convert to GPS84(地球坐标系), output coordinate WGS-84 accuracy within 1 to 2 meters.|
+|gcj_extract_wgs(double,double) -> json | GCJ02(火星坐标系) convert to GPS84, output coordinate WGS-84 accuracy within 0.5 meters. but compute cost more time than `gcj_to_wgs`. |
+
+> 关于互联网地图坐标系的说明见: [当前互联网地图的坐标系现状](https://github.com/aaronshan/hive-third-functions/tree/master/README-geo.md)
+
+
+### 9. url functions
+
+| function| description |
+|:--|:--|
+|url_encode(value) -> string | escapes value by encoding it so that it can be safely included in URL query parameter names and values|
+|url_decode(value) -> string | unescape the URL encoded value. This function is the inverse of `url_encode`. | 
+
 ## Use
 
 Put these statements into `${HOME}/.hiverc` or exec its on hive cli env.
@@ -118,7 +159,17 @@ create temporary function array_concat as 'cc.shanruifeng.functions.array.UDFArr
 create temporary function array_value_count as 'cc.shanruifeng.functions.array.UDFArrayValueCount';
 create temporary function array_slice as 'cc.shanruifeng.functions.array.UDFArraySlice';
 create temporary function array_element_at as 'cc.shanruifeng.functions.array.UDFArrayElementAt';
+create temporary function bit_count as 'cc.shanruifeng.functions.bitwise.UDFBitCount';
+create temporary function bitwise_and as 'cc.shanruifeng.functions.bitwise.UDFBitwiseAnd';
+create temporary function bitwise_not as 'cc.shanruifeng.functions.bitwise.UDFBitwiseNot';
+create temporary function bitwise_or as 'cc.shanruifeng.functions.bitwise.UDFBitwiseOr';
+create temporary function bitwise_xor as 'cc.shanruifeng.functions.bitwise.UDFBitwiseXor';
+create temporary function map_build as 'cc.shanruifeng.functions.map.UDFMapBuild';
+create temporary function map_concat as 'cc.shanruifeng.functions.map.UDFMapConcat';
+create temporary function map_element_at as 'cc.shanruifeng.functions.map.UDFMapElementAt';
+create temporary function map_equals as 'cc.shanruifeng.functions.map.UDFMapEquals';
 create temporary function day_of_week as 'cc.shanruifeng.functions.date.UDFDayOfWeek';
+create temporary function day_of_year as 'cc.shanruifeng.functions.date.UDFDayOfYear';
 create temporary function type_of_day as 'cc.shanruifeng.functions.date.UDFTypeOfDay'; 
 create temporary function zodiac_cn as 'cc.shanruifeng.functions.date.UDFZodiacSignCn';
 create temporary function zodiac_en as 'cc.shanruifeng.functions.date.UDFZodiacSignEn';
@@ -139,6 +190,14 @@ create temporary function id_card_birthday as 'cc.shanruifeng.functions.card.UDF
 create temporary function id_card_gender as 'cc.shanruifeng.functions.card.UDFChinaIdCardGender';
 create temporary function is_valid_id_card as 'cc.shanruifeng.functions.card.UDFChinaIdCardValid';
 create temporary function id_card_info as 'cc.shanruifeng.functions.card.UDFChinaIdCardInfo';
+create temporary function wgs_distance as 'cc.shanruifeng.functions.geo.UDFGeoWgsDistance';
+create temporary function gcj_to_bd as 'cc.shanruifeng.functions.geo.UDFGeoGcjToBd';
+create temporary function bd_to_gcj as 'cc.shanruifeng.functions.geo.UDFGeoBdToGcj';
+create temporary function wgs_to_gcj as 'cc.shanruifeng.functions.geo.UDFGeoWgsToGcj';
+create temporary function gcj_to_wgs as 'cc.shanruifeng.functions.geo.UDFGeoGcjToWgs';
+create temporary function gcj_extract_wgs as 'cc.shanruifeng.functions.geo.UDFGeoGcjExtractWgs';
+create temporary function url_encode as 'cc.shanruifeng.functions.url.UDFUrlEncode';
+create temporary function url_decode as 'cc.shanruifeng.functions.url.UDFUrlDecode';
 ```
 
 You can use these statements on hive cli env get detail of function.
@@ -166,6 +225,7 @@ Example:
 
 ```
 select day_of_week('2016-07-12') => 2
+select day_of_year('2016-01-01') => 1
 select type_of_day('2016-10-01') => 1
 select type_of_day('2016-07-16') => 2
 select type_of_day('2016-07-15') => 3
@@ -191,6 +251,13 @@ select array_slice(array(16,13,12,13,18,16,9,18), -2, 3) => [9,18]
 select array_element_at(array(16,13,12,13,18,16,9,18), -1) => 18
 ```
 
+```
+select map_build(array('key1','key2'), array(16,12)) => {"key1":16,"key2":12}
+select map_concat(map_build(array('key1','key2'), array(16,12)), map_build(array('key1','key3'), array(17,18))) => {"key1":17,"key2":12,"key3":18}
+select map_element_at(map_build(array('key1','key2'), array(16,12)), 'key1') => 16
+select map_equals(map_build(array('key1','key2'), array(16,12)), map_build(array('key1','key2'), array(16,12))) => true
+```
+
 ```
 select id_card_info('110101198901084517') => {"valid":true,"area":"东城区","province":"北京市","gender":"男","city":"北京市"}
 ```
@@ -215,3 +282,15 @@ select json_size('{"x": {"a": 1, "b": 2}}', '$.x'); => 2
 select json_size('{"x": [1, 2, 3]}', '$.x'); => 3
 select json_size('{"x": {"a": 1, "b": 2}}', '$.x.a'); => 0
 ```
+
+```
+select gcj_to_bd(39.915, 116.404) => {"lng":116.41036949371029,"lat":39.92133699351022}
+select bd_to_gcj(39.915, 116.404) => {"lng":116.39762729119315,"lat":39.90865673957631}
+select wgs_to_gcj(39.915, 116.404) => {"lng":116.41024449916938,"lat":39.91640428150164}
+select gcj_to_wgs(39.915, 116.404) => {"lng":116.39775550083061,"lat":39.91359571849836}
+select gcj_extract_wgs(39.915, 116.404) => {"lng":116.39775549316407,"lat":39.913596801757805}
+```
+
+```
+select url_encode('http://shanruifeng.cc/') => http%3A%2F%2Fshanruifeng.cc%2F
+```

pom.xml

@@ -6,7 +6,7 @@
 
     <groupId>cc.shanruifeng</groupId>
     <artifactId>hive-third-functions</artifactId>
-    <version>2.1.0</version>
+    <version>2.2.0</version>
 
     <properties>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@@ -23,7 +23,7 @@
         <dep.airlift.version>0.131</dep.airlift.version>
         <dep.jackson.version>2.4.4</dep.jackson.version>
         <dep.jmh.version>1.9.3</dep.jmh.version>
-        <dep.fastutil.version>6.5.9</dep.fastutil.version>
+        <junit.version>4.12</junit.version>
     </properties>
 
     <dependencyManagement>
@@ -83,9 +83,9 @@
             </dependency>
 
             <dependency>
-                <groupId>it.unimi.dsi</groupId>
-                <artifactId>fastutil</artifactId>
-                <version>${dep.fastutil.version}</version>
+                <groupId>junit</groupId>
+                <artifactId>junit</artifactId>
+                <version>${junit.version}</version>
             </dependency>
         </dependencies>
     </dependencyManagement>
@@ -139,8 +139,9 @@
         </dependency>
 
         <dependency>
-            <groupId>it.unimi.dsi</groupId>
-            <artifactId>fastutil</artifactId>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
         </dependency>
     </dependencies>
 

src/main/java/cc/shanruifeng/functions/array/UDFArrayConcat.java

@@ -35,7 +35,7 @@ public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumen
         // Check if two arguments were passed
         if (arguments.length != ARG_COUNT) {
             throw new UDFArgumentLengthException(
-                    "The function array_concat(array, array) takes exactly " + ARG_COUNT + "arguments.");
+                    "The function array_concat(array, array) takes exactly " + ARG_COUNT + " arguments.");
         }
 
         // Check if two argument is of category LIST

src/main/java/cc/shanruifeng/functions/array/UDFArrayContains.java

@@ -38,7 +38,7 @@ public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumen
         // Check if two arguments were passed
         if (arguments.length != ARG_COUNT) {
             throw new UDFArgumentLengthException(
-                    "The function array_contains(array, value) takes exactly " + ARG_COUNT + "arguments.");
+                    "The function array_contains(array, value) takes exactly " + ARG_COUNT + " arguments.");
         }
 
         // Check if ARRAY_IDX argument is of category LIST

src/main/java/cc/shanruifeng/functions/array/UDFArrayDistinct.java

@@ -1,6 +1,6 @@
 package cc.shanruifeng.functions.array;
 
-import it.unimi.dsi.fastutil.ints.IntArrays;
+import cc.shanruifeng.functions.fastuitl.ints.IntArrays;
 import java.util.ArrayList;
 import org.apache.hadoop.hive.ql.exec.Description;
 import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
@@ -38,7 +38,7 @@ public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumen
         // Check if two arguments were passed
         if (arguments.length != ARG_COUNT) {
             throw new UDFArgumentLengthException(
-                    "The function array_distinct(array) takes exactly " + ARG_COUNT + "arguments.");
+                    "The function array_distinct(array) takes exactly " + ARG_COUNT + " warguments.");
         }
 
         // Check if two argument is of category LIST

src/main/java/cc/shanruifeng/functions/array/UDFArrayElementAt.java

@@ -38,7 +38,7 @@ public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumen
         // Check if two arguments were passed
         if (arguments.length != ARG_COUNT) {
             throw new UDFArgumentLengthException(
-                    "The function array_element_at(array, index) takes exactly " + ARG_COUNT + "arguments.");
+                    "The function array_element_at(array, index) takes exactly " + ARG_COUNT + " arguments.");
         }
 
         // Check if ARRAY_IDX argument is of category LIST

src/main/java/cc/shanruifeng/functions/array/UDFArrayIntersect.java

@@ -1,6 +1,6 @@
 package cc.shanruifeng.functions.array;
 
-import it.unimi.dsi.fastutil.ints.IntArrays;
+import cc.shanruifeng.functions.fastuitl.ints.IntArrays;
 import java.util.ArrayList;
 import org.apache.hadoop.hive.ql.exec.Description;
 import org.apache.hadoop.hive.ql.exec.UDFArgumentException;

src/main/java/cc/shanruifeng/functions/array/UDFArrayJoin.java

@@ -46,7 +46,7 @@ public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumen
         if (arguments.length > MAX_ARG_COUNT || arguments.length < MIN_ARG_COUNT) {
             throw new UDFArgumentLengthException(
                     "The function array_join(array, delimiter) or array_join(array, delimiter, null_replacement) takes exactly "
-                            + MIN_ARG_COUNT + " or " + MAX_ARG_COUNT + "arguments.");
+                            + MIN_ARG_COUNT + " or " + MAX_ARG_COUNT + " arguments.");
         }
 
         // Check if ARRAY_IDX argument is of category LIST

src/main/java/cc/shanruifeng/functions/array/UDFArrayMax.java

@@ -1,6 +1,6 @@
 package cc.shanruifeng.functions.array;
 
-import it.unimi.dsi.fastutil.ints.IntArrays;
+import cc.shanruifeng.functions.fastuitl.ints.IntArrays;
 import org.apache.hadoop.hive.ql.exec.Description;
 import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
 import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;