merge develop

Author: bigerl

CMakeLists.txt

@@ -1,6 +1,6 @@
 cmake_minimum_required(VERSION 3.24)
-project(hypertrie VERSION 0.9.6
-        DESCRIPTION "A flexible data structure for low-rank, sparse tensors supporting slices by any dimension and einstein summation (einsum) and a flexible query interface")
+project(hypertrie VERSION 0.10.0
+        DESCRIPTION "The hypertrie is powering the Tentris triple store.")
 
 include(cmake/boilerplate_init.cmake)
 boilerplate_init()

README.md

@@ -1,38 +1,91 @@
-TODO: update readme
+# Hypertrie
 
-# hypertrie
+## Where is it used?
 
-A flexible data structure for low-rank, sparse tensors supporting slices by any dimension and einstein summation (
-einsum).
+- the backend of the [commercial](https://github.com/tentris/tentris)
+  and [research-prototype](https://github.com/dice-group/tentris-research-project) Tentris triple stores/RDF graph
+  databases
+- a sparse tensor representation that supports slicing by any dimension and einstein summation
+- a monolithic index that supports worst-case optimal joins (WCOJ) by providing all collation orders in a single
+  redundancy eliminating datastructure.
 
-For details on the data structure refer to https://tentris.dice-research.org/
+## What is it?
 
-## build
+Technically, a hypertrie stores $d$-tuples where $d$ is also called dimension (tensor) or depth (trie, index).
 
-### prerequisites
+It allows incremental slicing (tensor) by any dimension or select and project by any predicate (relational algebra).
+These properties are important to support worst-case optimal joins (WCOJ) efficiently.
 
-install conan, cmake and a C++20 compiler.
+## Asymptotic Guarantees
 
-and create a conan profile
+A depth-$d$ the hypertrie encoding a set of $z$ tuples requires at most $\mathcal O (z\cdot 2^{d-1}\cdot d)$ space. The
+runtime complexity of applying (inserting or deleting) a changeset set $\Delta$ of $d$-tuples to a depth-$d$ hypertrie
+is bound by the space complexity $\mathcal O (|\Delta| \cdot 2^{d-1}\cdot d)$ of a surrogate hypertrie that encodes the
+change set changeset $\Delta$.
 
- ```shell script
-conan profile new --detect default
-conan profile update settings.compiler.libcxx=libstdc++11 default
- ```
+# build
 
-You'll need some packages from DICE group's conan artifactory. Add it with:
+This is a template library. So there is nothing to build beyond tests.
 
-```shell script
-conan remote add dice-group https://conan.dice-research.org/artifactory/api/conan/tentris
+## prerequisites
+
+Software was tested on Ubuntu-22.04 with gcc-13 and clang-17, both using libstdc++-13 as C++ STL.
+
+Install with:
+
+```shell
+# gcc-13
+sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
+          
+# clang-17
+source /etc/os-release
+echo "deb http://apt.llvm.org/${UBUNTU_CODENAME}/ llvm-toolchain-${UBUNTU_CODENAME}-17 main" | sudo tee /etc/apt/sources.list.d/llvm-17.list
+curl https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/llvm.gpg > /dev/null
+# Ensure STL version
+sudo apt-get install -y libstdc++-13-dev
 ```
 
-### build
+Ensure cmake >3.24 is installed.
 
-```shell script
-mkdir build
-cd build
-conan install .. --build=missing
-cmake ..
+## Build the Tests
+
+Download `conan_provider.cmake` to the project directory:
+
+```shell
+wget https://github.com/conan-io/cmake-conan/raw/develop2/conan_provider.cmake -O conan_provider.cmake
+```
+
+(Optional) Some tests require libtorch. You can download it from https://pytorch.org/get-started/locally/ (works at
+least with Stable|Linux|LibTorch|C++|None). We tested with version 10.9.0. There have been reports about some newer
+versions being
+broken.
+
+Configure CMake (replace `path/to/libtorch` or remove the line before excuting).
+
+```shell
+cmake \
+-DCMAKE_PROJECT_TOP_LEVEL_INCLUDES="conan_provider.cmake" \
+-DBUILD_TESTING=On \
+-DLIBTORCH_PATH="path/to/libtorch" \
+-B build .
+```
+
+Build:
+
+```shell
+cmake --build . --parallel
+```
+
+Run:
+
+```shell
+ctest --parallel
+```
+
+If you want to exclude long-running validation tests, run:
+
+```shell
+ctest --parallel --exclude-regex "(tests_RawHypertrieContext_systematic)|(tests_RawHypertrieContext_systematic_metall)|(tests_HypertrieContext_systematic_metall)|(tests_Einsum)|(tests_Einsum_metall)"
 ```
 
 # running tests
@@ -45,12 +98,3 @@ make -j tests
 tests/tests
 ```
 
-Some tests are using [pytorch](https://github.com/pytorch/pytorch) which is not provided with the code.
-Those tests are disabled by default.
-To enable them, provide the path to the pytorch library via cmake variable `hypertrie_LIBTORCH_PATH`.
-Prebuild binaries may be downloaded via https://pytorch.org/get-started/locally/ (works at least with
-Stable|Linux|LibTorch|C++|None).
-
-```shell script
-cmake -DDBUILD_TESTING=ON -DLIBTORCH_PATH=/path/to/libtorch ..
-```

libs/einsum/src/dice/einsum/internal/operators/Operator.hpp

@@ -1,14 +1,14 @@
 #ifndef HYPERTRIE_OPERATOR_HPP
 #define HYPERTRIE_OPERATOR_HPP
 
+#include "dice/einsum/internal/operators/Operator_predeclare.hpp"
+
 #include "dice/einsum/internal/operators/CartesianOperator.hpp"
 #include "dice/einsum/internal/operators/CountOperator.hpp"
 #include "dice/einsum/internal/operators/EntryGeneratorOperator.hpp"
 #include "dice/einsum/internal/operators/JoinOperator.hpp"
 #include "dice/einsum/internal/operators/ResolveOperator.hpp"
 
-#include "dice/einsum/internal/operators/Operator_predeclare.hpp"
-
 
 #include <memory>
 #include <utility>

libs/hypertrie/CMakeLists.txt

@@ -3,26 +3,27 @@ set(lib_suffix "hypertrie")
 set(lib "${PROJECT_NAME}-${lib_suffix}")
 
 find_package(Threads REQUIRED)
-find_package(Boost REQUIRED COMPONENTS)
+find_package(Boost REQUIRED)
 find_package(robin_hood REQUIRED)
 find_package(dice-sparse-map REQUIRED)
 find_package(dice-hash REQUIRED)
 find_package(dice-template-library REQUIRED)
 
+# Define the library
 add_library(${lib} INTERFACE)
 add_library(${PROJECT_NAME}::${lib_suffix} ALIAS ${lib})
 
+target_include_directories(${lib} INTERFACE
+        $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src>)
+
 target_link_libraries(${lib} INTERFACE
         Threads::Threads
         Boost::headers
         robin_hood::robin_hood
         dice-hash::dice-hash
         dice-sparse-map::dice-sparse-map
         dice-template-library::dice-template-library
-        )
-
-target_include_directories(${lib} INTERFACE
-        $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src>)
+)
 
 include(${CMAKE_SOURCE_DIR}/cmake/install_components.cmake)
-install_component(INTERFACE ${lib_suffix} src)
+install_component(INTERFACE ${lib_suffix} src)

libs/hypertrie/src/dice/hypertrie.hpp

@@ -1,9 +1,9 @@
 #ifndef HYPERTRIE_HYPERTRIE_SI_HPP
 #define HYPERTRIE_HYPERTRIE_SI_HPP
 
-#include "dice/hypertrie/BulkInserter.hpp"
-#include "dice/hypertrie/HashJoin.hpp"
 #include "dice/hypertrie/Hypertrie.hpp"
+#include "dice/hypertrie/BulkUpdater.hpp"
+#include "dice/hypertrie/HashJoin.hpp"
 #include "dice/hypertrie/Hypertrie_version.hpp"
 
 #include "dice/hypertrie/Hypertrie_default_traits.hpp"