Dhf5.1.0 to dhf5.7.2 (#16)

Feature branch was originally started to upgrade the Data Hub Foundation version from 5.1.0 to 5.7.2 but when DH moved 5.8 during the changes, I removed all the DH components entirely. They are brought into the project with hubInit instead which is much more standard operating procedure and much cleaner.

* Add codepoint to match with created Element Range Index definitions.

* DH version bump to 5.7.2

* DH 5.1.0 to 5.7.2

* Gradle version 5.2.1 -> 6.9.1

* Remove DeployHubOtherServersCommand unit test refs

* mlDHFVersion no longer has any impact starting in version 5.3.0. You may safely remove this from your properties file.

* runAllPPM -> examples:person-position-matching:runAll

* Add range-element-index instructions.

* Comment out unit test framework.

* Remove instruction re: data zip file which longer exists nor is necessary

* runAll -> runAndDeployAll

* Close bracket for cleanliness

* Formatting, HADOOP_HOME, and remove Windows drug load instruction.

* Fix -input_file_path for drugZipLoad. Clarify instructions.

* Update loadPubMedCentral instructions to include FTP site and path location.

* Adjust -input_file_path of pubMedInputFlowFTP. loadStudy -> LoadStudy

* loadDrug -> LoadDrug flow-name change.

* Internal grove-core-react-redux-containers replaced with public github accessible version.

* Deleted all of the DH hubInit generated files.

* Update versions, repos, and remove DH compile.

* Changed the instructions to initialize Data Hub with hubInit. Formatting.

* Remove deprecated jcenter.

* Configure example specific indexes using mlConfigPaths instead of modifying final-database.json directly.

Author: derekbrouwer

README.md

@@ -11,7 +11,6 @@ There are two main steps to use this application:
 ## 1. Install Research Hub
 #### Prerequites
 1. MarkLogic 9.0-11 or 10.0-3
-2. DataHub 5.1.0 (included)
 3. Java 8+ (used for running QuickStart and gradle-wrapper)
 4. Node 10 or 12
 5. Optionally Docker Desktop if you want/need to run MarkLogic inside a container
@@ -22,12 +21,18 @@ It is recommended to use the default ports, DB names, and App Server names defin
 
 #### 1.2 Deploy DHF
 
-    ./gradlew mlDeploy
+First, generate Data Hub source files:
+
+    `./gradlew hubInit`
+
+Second, deploy the database components (including Data Hub) to MarkLogic:
+
+    `./gradlew mlDeploy`
 
 ### 1.3 Install UI
 
-    cd ui
-    npm install
+    `cd ui`
+    `npm install`
 
 ## 2. Install One of the Included Examples
 Two examples are included in this code base. By copying these example files into the main code area, and modifying some core files, you will be installing a fully-functional Research Hub. These are included to provide sample code and patterns for building your own research hubs.
@@ -42,37 +47,39 @@ To install one of the example domains, follow the README.md file in that example
 You can create your own entity by running a gradle command to scaffold the files you need.
 
 run
-./gradlew createEntity -PentityName=YourEntityName
+`./gradlew createEntity -PentityName=YourEntityName`
 
 The task will create the necessary files and print out their locations for you to go edit.
 
 ### Deleting a scaffolded entity
 If for some reason you need to delete an entity that you scaffolded, simply run this command.
 run
 
-./gradlew deleteEntity -PentityName=YourEntityName
+`./gradlew deleteEntity -PentityName=YourEntityName`
 
 ### Building your own concept
 You can create your own concept by running a gradle command to scaffold the files you need.
 
 run
-./gradlew createConcept -PconceptName=YourConceptName
+`./gradlew createConcept -PconceptName=YourConceptName`
 
 The task will create the necessary files and print out their locations for you to go edit.
 
 ### Deleting a scaffolded concept
 If for some reason you need to delete a concept that you scaffolded, simply run this command.
 run
 
-./gradlew deleteConcept -PconceptName=YourConceptName
+`./gradlew deleteConcept -PconceptName=YourConceptName`
 
 The task will create the necessary files and print out their locations for you to go edit.
 
 ## Running the UI
 After installing Research Hub and a domain, you can start the ui using:
 
-    cd ui
-    npm start
+    `cd ui`
+    `npm start`
+
+If the start fails, it may be necessary to install the nodemon (Node Monitor), cross-env, and concurrently packages with `npm install nodemon`, `npm install cross-env`, and `npm install concurrently` respectively from the ui directory. Typically, these are installed with `-g` option which makes the packages globally available across all projects.
 
 ## Running Tests
 After installing Research Hub and a domain, you can run any tests that were created by going to http://localhost:8015/test/ (or http://localhost:8115/test/ if using our Docker setup), and using the UI there to run tests.

build.gradle

@@ -2,12 +2,19 @@ import java.text.SimpleDateFormat;
 
 buildscript {
   repositories {
-     maven {url 'http://developer.marklogic.com/maven2/'}
-     mavenLocal()
-     jcenter()
+		// To use gradle in disconnected mode, you just need to set the 'disconnected' property. E.g. gradle compileJava -Pdisconnected
+		if (project.hasProperty("disconnected") && !"FALSE".equalsIgnoreCase(disconnected)) {
+			println "Using offline buildscript dependency repositories"
+			maven { url uri(projectMavenRepo) }
+		} else {
+			println "Using online buildscript dependency repositories"
+      maven {url 'https://developer.marklogic.com/maven2/'}
+      mavenLocal()
+      mavenCentral()
+    }
   }
   dependencies {
-    classpath "com.marklogic:marklogic-unit-test-client:1.0.0"
+    classpath "com.marklogic:marklogic-unit-test-client:1.2.1"
   }
 }
 
@@ -24,20 +31,28 @@ plugins {
     // See https://github.com/stevesaliman/gradle-properties-plugin
     // specify the env on the command line with:
     // gradle -PenvironmentName=x ...
-    id 'net.saliman.properties' version '1.4.6'
+    id 'net.saliman.properties' version '1.5.1'
 
     // This gradle plugin extends the ml-gradle plugin with
     // commands that make the Data Hub Framework do its magic
-    id 'com.marklogic.ml-data-hub' version '5.1.0'
+    id 'com.marklogic.ml-data-hub' version '5.8.0'
 }
 
 repositories {
-    jcenter()
-    maven {url 'http://developer.marklogic.com/maven2/'}
+	// To use gradle in disconnected mode, you just need to set the 'disconnected' property. E.g. gradle compileJava -Pdisconnected
+	if (project.hasProperty("disconnected") && !"FALSE".equalsIgnoreCase(disconnected)) {
+		println "Using offline dependency repositories"
+		maven { url uri(projectMavenRepo) }
+	} else {
+		println "Using online dependency repositories"
+		mavenCentral()
+		maven { url "https://developer.marklogic.com/maven2/" }
+	}
 }
 
 configurations {
-    mlcp
+  corb
+  mlcp
 }
 
 dependencies {
@@ -46,8 +61,8 @@ dependencies {
         mlBundle "com.marklogic:marklogic-unit-test-modules:1.0.0"
     }
 
-    compile 'com.marklogic:marklogic-data-hub:5.1.0'
-    mlcp "com.marklogic:mlcp:10.0.2"
+    corb "com.marklogic:marklogic-corb:2.4.1"
+    mlcp "com.marklogic:mlcp:10.0.6.1"
 
     if ("true".equalsIgnoreCase(project.findProperty("isDeployUnitTestFramework"))) {
       testCompile "com.marklogic:marklogic-unit-test-client:1.0.beta"
@@ -87,13 +102,14 @@ def buildTime() {
     return df.format(new Date())
 }
 
-// Database client config for mlUnitTest
+/* Database client config for mlUnitTest
 ext {
   mlUnitTest.databaseClientConfig.host = mlHost
   mlUnitTest.databaseClientConfig.port = mlTestFinalPort.toInteger()
   mlUnitTest.databaseClientConfig.username = mlUsername
   mlUnitTest.databaseClientConfig.password = mlPassword
 }
+/*
 
 /**
  * Delete a collection in the Final Database.
@@ -149,30 +165,6 @@ task createTestDirs {
   }
 }
 
-if ("true".equalsIgnoreCase(project.findProperty("isDeployUnitTestFramework"))) {
-  (new File(buildDir, "/test-results/marklogic-unit-tests")).mkdirs()
-  def testStagingDbFile = new File(buildDir, "test-staging-database.json")
-  testStagingDbFile.write(new File("src/main/hub-internal-config/databases/staging-database.json").text)
-
-  def testFinalDbFile = new File(buildDir, "test-final-database.json")
-  testFinalDbFile.write(new File("src/main/ml-config/databases/final-database.json").text)
-
-  def newCommands = new ArrayList<>()
-  for (def c : mlAppDeployer.commands) {
-    newCommands.add(c)
-    if (c instanceof com.marklogic.appdeployer.command.databases.DeployOtherDatabasesCommand) {
-      newCommands.add(new DeployHubTestDatabaseCommand(hubConfig, testStagingDbFile, "staging-database.json", mlStagingDbName + "-TEST"))
-      newCommands.add(new DeployHubTestDatabaseCommand(hubConfig, testFinalDbFile, "final-database.json", mlFinalDbName + "-TEST"))
-    }
-    // Loading roles in two phases breaks the preview feature, so it's disabled
-    else if (c instanceof com.marklogic.hub.deploy.commands.DeployHubOtherServersCommand) {
-      newCommands.add(new DeployHubTestServerCommand(dataHub, "staging-server.json", mlStagingAppserverName + "-TEST", mlTestStagingPort.toInteger(), mlStagingDbName + "-TEST"))
-      newCommands.add(new DeployHubTestServerCommand(dataHub, "final-server.json", mlFinalAppserverName + "-TEST", mlTestFinalPort.toInteger(), mlFinalDbName + "-TEST"))
-    }
-  }
-  mlAppDeployer.commands = newCommands
-}
-
 task hubUndeployTestResources(type: com.marklogic.gradle.task.MarkLogicTask) {
     description = "Undeploys the test servers and databases that were created via hubDeployTestResources"
     doLast {
@@ -222,30 +214,6 @@ class DeployHubTestDatabaseCommand extends com.marklogic.hub.deploy.commands.Dep
     }
 }
 
-class DeployHubTestServerCommand extends com.marklogic.hub.deploy.commands.DeployHubOtherServersCommand {
-    String serverName
-    int port
-    String contentDatabaseName
-
-    DeployHubTestServerCommand(DataHub dataHub, String serverFilenamePattern, String serverName, int port, String contentDatabaseName) {
-        super(dataHub)
-        setResourceFilenamesIncludePattern(Pattern.compile(serverFilenamePattern))
-        this.serverName = serverName
-        this.port = port
-        this.contentDatabaseName = contentDatabaseName
-    }
-
-    @Override
-    protected String copyFileToString(File f) {
-        String payload = super.copyFileToString(f)
-        ObjectNode node = new ObjectMapper().readTree(payload)
-        node.set("server-name", new TextNode(serverName))
-        node.set("port", new TextNode(port + ""))
-        node.set("content-database", new TextNode(contentDatabaseName))
-        return node.toString()
-    }
-}
-
 task clearStagingDatabase(type: com.marklogic.gradle.task.databases.ClearDatabaseTask) {
   doFirst {
     project.ext.database = project.properties.mlStagingDbName

buildSrc/build.gradle

@@ -2,7 +2,6 @@ apply plugin: 'groovy'
 
 repositories {
   mavenCentral()
-  jcenter()
 }
 
 dependencies {

examples/hr360/README.md

@@ -65,6 +65,15 @@ And add an import for `Skill`, and include it in the `export` section so it look
     workspace
   }
 
+### Adding necessary indexes
+
+Add to /gradle.properties a line to merge hr360 indexes with the base final database configuration:
+
+  `mlConfigPaths=src/main/ml-config,examples/hr360/src/main/ml-config`
+
+This line will include the indexes defined in the `examples\hr360\src\main\ml-config\databases\final-database.json` file to the base version of the final database defined in `src\main\ml-config\databases\final-database.json`.
+
+
 ### Build and deploy the changes
 
 Use this commands to deploy all the changes and code to the backend:

examples/hr360/src/main/ml-config/databases/final-database.json

@@ -0,0 +1,61 @@
+{
+  "database-name": "%%mlFinalDbName%%",
+  "range-element-index": [
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "requiredCompetency",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    },
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "departmentName",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    },
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "location",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    },
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "reasonType",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    },
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "category",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    },
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "languageSkills",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    },
+    {
+      "scalar-type": "string",
+      "namespace-uri": "",
+      "localname": "toolSkills",
+      "collation": "http://marklogic.com/collation/codepoint",
+      "range-value-positions": false,
+      "invalid-values": "reject"
+    }
+  ]
+}

examples/hr360/src/main/ml-modules/options/employee.xml

@@ -97,23 +97,23 @@
     </range>
   </constraint>
   <constraint name="LanguageSkill">
-    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/" >
+    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/codepoint" >
       <facet-option>limit=10</facet-option>
       <facet-option>frequency-order</facet-option>
       <facet-option>descending</facet-option>
       <element ns="" name="languageSkills"/>
     </range>
   </constraint>
   <constraint name="ToolSkill">
-    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/" >
+    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/codepoint" >
       <facet-option>limit=10</facet-option>
       <facet-option>frequency-order</facet-option>
       <facet-option>descending</facet-option>
       <element ns="" name="toolSkills"/>
     </range>
   </constraint>
   <constraint name="ProductSkill">
-    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/" >
+    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/codepoint" >
       <facet-option>limit=10</facet-option>
       <facet-option>frequency-order</facet-option>
       <facet-option>descending</facet-option>

examples/hr360/src/main/ml-modules/options/jobopening.xml

@@ -89,7 +89,7 @@
   </constraint>
  -->
   <constraint name="RequiredCompetency">
-    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/" >
+    <range type="xs:string" facet="true" collation="http://marklogic.com/collation/codepoint" >
       <facet-option>limit=10</facet-option>
       <facet-option>frequency-order</facet-option>
       <facet-option>descending</facet-option>