Hpcc 19200 ssl#1
Open
rpastrana wants to merge 45 commits into
Open
Conversation
#Spark-HPCC-3 Set java level and wsclient dep Reviewed-By: John Holt <john.d.holt@lexisnexis.com> Reviewed-By: Michael Gardner <michael.gardner@lexisnexis.com>
HpccFile changed to get compressed flag from the DFUFileDetail and pass on to FilePart constructor. FilePart changed to carry compressed flag and to change file parts place holder to number of parts (HPCC-19619). PlainConnection changed to process error message strings without length prefix; and to add compredded flag to json request string. Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
HpccFile changed to get compressed flag from the DFUFileDetail and pass on to FilePart constructor. FilePart changed to carry compressed flag and to change file parts place holder to number of parts (HPCC-19619). PlainConnection changed to process error message strings without length prefix; and to add compredded flag to json request string. Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
Signed-off-by: Michael Gardner <michael.gardner@lexisnexisrisk.net>
- Adds apache 2 2018 license headers - Organizes import statements - Removes author entries Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
hpcc-spark pom.xml changes Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: John Holt <john.d.holt@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
…essed HPCC-19611 Compressed file read Reviewed-By: John Holt <john.d.holt@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-19758 Provide copyright header and cleanup Reviewed-By: John Holt <john.d.holt@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Changed HpccFile to accept a field list string as a selection parameter. List is comma separated with compound names in normal dot notation. Changed RecordDef to have the original JSON data definition changed. Changed the DefToken to convert back to string format. Added ColumnPruner, TargetColumn, and DefEntry et.al. to implement pruning fields and types from the output definition. Updated test programs to solicit a field selection string. Simplified RDDTest to only show data on console. Applied changes from review comments to ColumnPruner, HpccFile, TargetColumn, the DefEntry_ classes. Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
HPCC-19201 Support field selection to prune unwanted data Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Reads index datasets with filtering. Increased the number of datatypes supported and suppressed datatypes that should not be supported. Indexes have an additional file part, so the relationship between Spark partitions and file parts was made indirect. ColumnPruner was enhanced to suppress datatypes that are unsupported and to change the output format of particular types (like QSTRING) into a more standard type. The enhancement eliminated shortcut processing through the ColumnPruner. HpccDataFrameFactory was modified to use the replacement of FilePart with HpccPart. HpccFile was enhanced by adding several convenience constructors and by adding the display of the JSON layout definition in the event of a problem processing the definition. The FilePart object was refactored into an HpccPart object employed by HpccFile and a DataPart object. HpccPart is the result of refactoring FilePart to make the relationship between Spark partitions and Thor file partitions indirect. HpccRDD was modified to use the replacement of FilePart with HpccPart. HpccRemoteFileReader was modified to use the new HpccPart. package-info updated to reflect the change in class name of FilePart to HpccPart. AddrRemapper was enhanced to work in cases where there are more file parts than nodes. Cleaned up the implementation to get a list of the unique IP values instead of assuming that the IP values were unique. BinaryRecordReader was modified to use the DataPart object that was refactored from the FilePart. ClusterRemapper was modified to remove the number of parts. DataPartition was created by refactoring FilePart. DefEntry removed unused constructor and added tracking of type, length, and type flag. The short cut processing for the no pruning case was removed. DefEntryField was modified to track if the definition should be suppressed; or if the definition had been modified. An unused empty constructor was removed. DefEntryRoot was modified to remove an unused constructor and to track suppressions and type changes. DefEntryType was modified to remove an unused constructor and to track suppression and type changes. DefToken was enhanced by adding comments and a new constructor used for the update of type and length information. FieldFilter class was added to capture filter criteria for a specified field name. the constructor takes FieldFilterRanges. FieldFilterRange class capture the properties for a filter range. FileFilter class was added to capture filter information from either an array of FieldFilter objects or a string version of the filter expression. PlainConnection updated for the FilePart refactor. Updated to request an index read instead of a disk read when the file is an index, and updated to add the field filter to the read request. RemapInfo was updated to use zero nodes for the empty constructor. TypeDef was enhanced to support type revision and suppression. DataframeTest was updated for the FilePart refactor. HpccFiletest was updated for the FilePart refactor and to add file filtering and to allow the user to specify which part to read. RDDTest was updated for the FilePart refactor. REcordTest was updated to support Field filters and the FilePart refactor.
Added fix for changed protocol on continuation command. The name 'cursor' was changed to 'handle' in protocol. Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
HPCC-19375 Index pull into Spark environment Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Defines stream read command value - Inserts stream reaad command on all requests - Reads 4 byte status - Outputs status error value in hex format - Utilizes static HPCC charset value Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
…tNewProtocol HPCC-19284 Implement new dafileserv protocol Reviewed-By: John Holt <john.d.holt@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Extended HpccPart to accept an array of DataPartitions, each DataPartition mapped to a singler THOR file pert. Extended HpccRemoteFileReader to work with an array of DataPartitions. Extended test code for multiple data partitions. Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
Extended HpccPart to accept an array of DataPartitions, each DataPartition mapped to a singler THOR file pert. Extended HpccRemoteFileReader to work with an array of DataPartitions. Extended test code for multiple data partitions. Signed-off-by: johnholt <john.d.holt@lexisnexisrisk.com>
HPCC-19604 Spark access to super files Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Adds field name to data converting error message Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
…rrMsg HPCC-20108 Improve error message Reviewed-By: James McMullan <james.mcmullan@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Replaced Content class hierarchy with native Java & Spark types - Replaced Record and HPCCRow classes with builtin Spark GenericRowWithSchema class - Organized imports Signed-off-by: James McMullan <James.McMullan@lexisnexis.com>
HPCC-19370 Eliminate Content classes, Record class and HPCCRow class Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Register / use builtin Python picklers with GenericRowWithSchema - Implemented a getRDD version that allows PySpark to create RDDs Signed-off-by: James McMullan <James.McMullan@lexisnexis.com>
- Removes restriction due to change in ESP HPCC-20071 Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
HPCC-19369 Pyrolite serialization and de-serialization Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Added replyLimit parameter to read requests - Determined 4MB requests to be optimal based on benchmarking Signed-off-by: James McMullan James.McMullan@lexisnexis.com
SPAR-7 Set response size on read requests Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
…perfilewmultiplesubfiles HPCC-20039 Remove multi-subfile restriction on superfiles Reviewed-By: James McMullan <james.mcmullan@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Added decimal support - Fixed signed integer loading - Added unsigned integer support - Added varstring support - Added data support - Fixed set loading - Added support for all scalar set types - Merged code for set & datasets - Fixed dataset loading - Improved metadata structure - Removed unused class ParsedContent - Added fix for variable length qstrings - Renamed MISSING enum to UNKNOWN - Replaced various magic numbers with named constant Signed-off-by: James McMullan <James.McMullan@lexisnexis.com>
HPCC-20217 Spark-HPCC Datatype Loading Issues Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Acquires file access artifact as part of file stream process - Changes dafileserv request to new secure format - Use readfully rather than reading err message byte by byte - Resctructure error message Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
…Access HPCC-19203 Utilize DAFILESRV file access mechanism Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
SPARK-HPCC 7.0.0 Branch Reviewed-By: Michael Gardner <michael.gardner@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Creates RFCCodes class with all known dafilesrv codes - Handles 2 file access expiry error codes - Allows developers to overwrite default expiry Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
- Fix indent issues Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
…Codes HPCC-20580 Track DAFILESERV trx codes Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Modified BinaryRecordReader to not create the additional Row - Refactored how record parsing returns results - Improved error messages Signed-off-by: James McMullan James.McMullan@lexisnexis.com
HPCC-20582 BinaryRecordReader creates an additional enclosing Row Reviewed-By: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Introduces connection timeout Defaulted to 1000mS - Iterates over all available copy ips until connection is established - Utilizes file part information from new fileaccess wsdfu method - Restructure FilePart to only contain 1 filepartition - Does not drill down to subfiles for filepart info - Utilize wsclient soap response wrappers - Removes HPCCPart - Replaces with Datapartition - Adds new reviceips signature for convenience Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
…ilepartcopies Hpcc 20154 Spark-hpcc should utilize all file part copies Reviewed-By: James McMullan <james.mcmullan@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Removes portremapper - Restructures cluster remapper to provide RS port and ssl use - Removes concept of clear and ssl ports Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
HPCC-20555 Consume rowservice connection information Reviewed-By: James McMullan <james.mcmullan@lexisnexis.com> Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
- Chooses to perform ssl based connection based on info from ESP Signed-off-by: Rodrigo Pastrana <rodrigo.pastrana@lexisnexis.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.