1.4.2-beta.2

Additions

• Added GenericNeuralStack[P, I, T] as a base class for Neural Stack API
• Added LazyNeuralStack[P, I] where the layers are lazily spawned.

1.4.2-beta.1

Additions

Package dynaml.models.neuralnets

• Added GenericAutoEncoder[LayerP, I], the class AutoEncoder is now deprecated

Package dynaml.kernels

• Added ScaledKernel[I] representing kernels of scaled Gaussian Processes.

Package dynaml.models.bayes

• Added * method to GaussianProcessPrior[I, M] which creates a scaled Gaussian Process prior using the newly minted ScaledKernel[I] class

Improvements

• Fixed issue with creation of MeasurableFunction instances from RandomVariable instances
• Added Kronecker structure speed up to energy (marginal likelihood) calculation of multi-output GP models

1.4.1

Pipes API

Additions

The pipes API has been vastly extended by creating pipes which encapsulate functions of multiple arguments leading to the following end points.

• DataPipe2[A, B, C]: Pipe which takes 2 arguments
• DataPipe3[A, B, C, D] : Pipe which takes 3 arguments
• DataPipe4[A, B, C, D, E]: Pipe which takes 4 arguments

Furthermore there is now the ability to create pipes which return pipes, something akin to curried functions in functional programming.

• MetaPipe: Takes an argument returns a DataPipe
• MetaPipe21: Takes 2 arguments returns a DataPipe
• MetaPipe12: Takes an argument returns a DataPipe2

A new kind of Stream data pipe, StreamFlatMapPipe is added to represent data pipelines which can perform flat map like operations on streams.

val mapFunc: (I) => Stream[J] = ...
val streamFMPipe = StreamFlatMapPipe(mapFunc)

• Added Data Pipes API for Apache Spark RDDs.

val num = 20
val numbers = sc.parallelize(1 to num)
val convPipe = RDDPipe((n: Int) => n.toDouble)

val sqPipe = RDDPipe((x: Double) => x*x)

val sqrtPipe = RDDPipe((x: Double) => math.sqrt(x))

val resultPipe = RDDPipe((r: RDD[Double]) => r.reduce(_+_).toInt)

val netPipeline = convPipe > sqPipe > sqrtPipe > resultPipe
netPipeline(numbers)

• Added UnivariateGaussianScaler class for gaussian scaling of univariate data.

Core API

Additions

Package dynaml.models.bayes

This new package will house stochastic prior models, currently there is support for GP and Skew GP priors, to see a starting example see stochasticPriors.sc in the scripts directory of the DynaML source.

---

Package dynaml.kernels

• Added evaluateAt(h)(x,y) and gradientAt(h)(x,y); expressing evaluate(x,y) and gradient(x,y) in terms of them
• Added asPipe method for Covariance Functions
• For backwards compatibility users are advised to extend
  LocalSVMKernel in their custom Kernel implementations incase they do
  not want to implement the evaluateAt API endpoints.
• Added FeatureMapKernel, representing kernels which can be explicitly decomposed into feature mappings.
• Added Matern half integer kernel GenericMaternKernel[I]
• Added block(S: String*) method to block any hyper-parameters of kernels.
• Added NeuralNetworkKernel and GaussianSpectralKernel.
• Added DecomposableCovariance

import io.github.mandar2812.dynaml.DynaMLPipe._
import io.github.mandar2812.dynaml.kernels._


implicit val ev = VectorField(6)
implicit val sp = breezeDVSplitEncoder(2)
implicit val sumR = sumReducer

val kernel = new LaplacianKernel(1.5)
val other_kernel = new PolynomialKernel(1, 0.05)

val decompKernel = new DecomposableCovariance(kernel, other_kernel)(sp, sumReducer)

val other_kernel1 = new FBMKernel(1.0)

val decompKernel1 = new DecomposableCovariance(decompKernel, other_kernel1)(sp, sumReducer)

val veca = DenseVector.tabulate[Double](8)(math.sin(_))
val vecb = DenseVector.tabulate[Double](8)(math.cos(_))

decompKernel1.evaluate(veca, vecb)

---

Package dynaml.algebra

Partitioned Matrices/Vectors and the following operations

• Addition, Subtraction
• Matrix, vector multiplication
• LU, Cholesky
• A\y, A\Y

Added calculation of quadratic forms, namely:

• quadraticForm which calculates x^T A^-1 x
• crossQuadraticForm which calculates y^T A^-1 x

Where A is assumed to be a symmetric positive semi-definite matrix

Usage:

import io.github.mandar2812.dynaml.algebra._

val x: DenseVector[Double] = ...
val y: DenseVector[Double] = ...
val a: DenseMatrix[Double] = ...

quadraticForm(a,x)
crossQuadraticForm(y, a, x)

---

Package dynaml.modelpipe

New package created, moved all inheriting classes of ModelPipe to this package.

Added the following:

• GLMPipe2 A pipe taking two arguments and returning a GeneralizedLinearModel instance
• GeneralizedLeastSquaresPipe2:
• GeneralizedLeastSquaresPipe3:

---

Package dynaml.models

• Added a new Neural Networks API: NeuralNet and GenericFFNeuralNet, for an example refer to TestNNDelve in dynaml-examples.
• GeneralizedLeastSquaresModel: The GLS model.
• ESGPModel: The implementation of a skew gaussian process regression model
• Warped Gaussian Process models WIP
• Added mean function capability to Gaussian Process and Student T process models.
• Added Apache Spark implementation of Generalized Linear Models; see SparkGLM, SparkLogisticModel, SparkProbitGLM

---

Package dynaml.probability

• MultivariateSkewNormal as specified in Azzalani et. al
• ExtendedMultivariateSkewNormal
• UESN and MESN representing an alternative formulation of the skew gaussian family from Adcock and Shutes.
• TruncatedGaussian: Truncated version of the Gaussian distribution.
• Matrix Normal Distribution
• Added Expectation operator for RandomVariable implementations in the io.github.mandar2812.dynaml.probability package object. Usage example given below.
• SkewGaussian, ExtendedSkewGaussian: An breeze implementation of the SkewGaussian and extended Skew-Gaussian distributions respectively
• PushforwardMap, DifferentiableMap added: PushforwardMap enables creating new random variables with defined density from base random variables.

import io.github.mandar2812.dynaml.analysis._
import io.github.mandar2812.dynaml.probability._
import io.github.mandar2812.dynaml.probability.distributions._

val g = GaussianRV(0.0, 0.25)
val sg = RandomVariable(SkewGaussian(1.0, 0.0, 0.25))

//Define a determinant implementation for the Jacobian type (Double in this case)
implicit val detImpl = identityPipe[Double]

//Defines a homeomorphism y = exp(x) x = log(y)
val h: PushforwardMap[Double, Double, Double] = PushforwardMap(
  DataPipe((x: Double) => math.exp(x)),
  DifferentiableMap(
    (x: Double) => math.log(x),
    (x: Double) => 1.0/x)
)

//Creates a log-normal random variable
val p = h->g

//Creates a log-skew-gaussian random variable
val q = h->sg

//Calculate expectation of q
println("E[Q] = "+E(q))

• Added Markov Chain Monte Carlo (MCMC) based inference schemes ContinuousMCMC and the underlying sampling implementation in GeneralMetropolisHastings.
• Added implementation of Approximate Bayesian Computation (ABC) in the ApproxBayesComputation class.

//The mean
val center: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among rows
val sigmaRows: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among columns
val sigmaCols: DenseMatrix[Double] = ...
val matD = MatrixNormal(center, sigmaRows, sigmaCols)

• Matrix T Distribution (Experimental)

//The degrees of freedom (must be > 2.0 for existence of finite moments)
val mu: Double = ...
//The mean
val center: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among rows
val sigmaRows: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among columns
val sigmaCols: DenseMatrix[Double] = ...
val matD = MatrixT(mu, center, sigmaCols, sigmaRows)

---

Package dynaml.optimization

• Added ProbGPCommMachine which performs grid search or CSA and then instead of selecting a single hyper-parameter configuration calculates a weighted Gaussian Process committee where the weights correspond to probabilities or confidence on each model instance (hyper-parameter configuration).

Package dynaml.utils

• Added multivariate gamma function

//Returns logarithm of multivariate gamma function
val g = mvlgamma(5, 1.5)

---

Package dynaml.dataformat

• Added support for reading MATLAB .mat files in the MAT object.

Improvements/Bug Fixes

Package dynaml.probability

• Removed ProbabilityModel and replaced with JointProbabilityScheme and BayesJointProbabilityScheme, major refactoring to RandomVariable API.

---

Package dynaml.optimization

• Improved logging of CoupledSimulatedAnnealing
• Refactored GPMLOptimizer to GradBasedGlobalOptimizer

---

Package dynaml.utils

• Correction to utils.getStats method used for calculating mean and variance of data sets consisting of DenseVector[Double].
• minMaxScalingTrainTest minMaxScaling of DynaMLPipe using GaussianScaler ins...

v1.4.1-beta.13

Additions

Pipes API

The pipes API has been vastly extended by creating pipes which encapsulate functions of multiple arguments leading to the following end points.

• DataPipe2[A, B, C]: Pipe which takes 2 arguments
• DataPipe3[A, B, C, D] : Pipe which takes 3 arguments
• DataPipe4[A, B, C, D, E]: Pipe which takes 4 arguments

Furthermore there is now the ability to create pipes which return pipes, something akin to curried functions in functional programming.

• MetaPipe: Takes an argument returns a DataPipe
• MetaPipe21: Takes 2 arguments returns a DataPipe
• MetaPipe12: Takes an argument returns a DataPipe2

Core API

Package dynaml.algebra

Added calculation of quadratic forms, namely:

• quadraticForm which calculates x^T A^-1 x
• crossQuadraticForm which calculates y^T A^-1 x

Where A is assumed to be a symmetric positive semi-definite matrix

Usage:

import io.github.mandar2812.dynaml.algebra._

val x: DenseVector[Double] = ...
val y: DenseVector[Double] = ...
val a: DenseMatrix[Double] = ...

quadraticForm(a,x)
crossQuadraticForm(y, a, x)

Package dynaml.modelpipe

New package created, moved all inheriting classes of ModelPipe to this package.

Added the following:

• GLMPipe2 A pipe taking two arguments and returning a GeneralizedLinearModel instance
• GeneralizedLeastSquaresPipe2:
• GeneralizedLeastSquaresPipe3:

Package dynaml.models

• GeneralizedLeastSquaresModel: The GLS model.
• AbstractSkewGPModel: The implementation of a skew gaussian process regression model

Package dynaml.probability

• MultivariateSkewNormal as specified in Azzalani et. al
• ExtendedMultivariateSkewNormal
• UESN and MESN representing an alternative formulation of the skew gaussian family from Adcock and Shutes.
• TruncatedGaussian: Truncated version of the Gaussian distribution.

Improvements

Core API

Package dynaml.optimization

• Improved logging of CoupledSimulatedAnnealing

1.4.1-beta.12

Improvements

Core API

Package dynaml.optimization

• Increased verbosity of logging in CoupledSimulatedAnnealing

Additions

Pipes API

• Added StreamFlatMapPipe to create data pipelines which can perform flat map like operations on streams.

val mapFunc: (I) => Stream[J] = ...
val streamFMPipe = StreamFlatMapPipe(mapFunc)

Core API

Package dynaml.kernels

• Added evaluateAt(h)(x,y) and gradientAt(h)(x,y); expressing
  evaluate(x,y) and gradient(x,y) in terms of them
• Added asPipe method for Covariance Functions
• For backwards compatibility users are advised to extend
  LocalSVMKernel in their custom Kernel implementations incase they do
  not want to implement the evaluateAt API endpoints.

Package dynaml.optimization

• Added ProbGPCommMachine which performs grid search or CSA and then instead of selecting a single hyper-parameter configuration calculates a weighted Gaussian Process committee where the weights correspond to probabilities or confidence on each model instance (hyper-parameter configuration).

Package dynaml.probability.distributions

• Matrix Normal Distribution

//The mean
val center: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among rows
val sigmaRows: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among columns
val sigmaCols: DenseMatrix[Double] = ...
val matD = MatrixNormal(center, sigmaRows, sigmaCols)

• Matrix T Distribution (Experimental)

//The degrees of freedom (must be > 2.0 for existence of finite moments)
val mu: Double = ...
//The mean
val center: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among rows
val sigmaRows: DenseMatrix[Double] = ...
//Covariance (positive semi-def) matrix among columns
val sigmaCols: DenseMatrix[Double] = ...
val matD = MatrixT(mu, center, sigmaCols, sigmaRows)

Package dynaml.utils

• Added multivariate gamma function

//Returns logarithm of multivariate gamma function
val g = mvlgamma(5, 1.5)

1.4.1-beta.11

Bug Fix Release

Fixes

• Fix to CoRegCauchyKernel: corrected mismatch of hyper-parameter string
• Fix to SVMKernel objects matrix gradient computation in the case when kernel dimensions are not multiples of block size.

Improvements

• Logging of GridSearch

1.4.1-beta.10

Additions

Models

• Warped Gaussian Process models WIP
• Added mean function capability to Gaussian Process and Student T process models.
• Added Apache Spark implementation of Generalised Linear Models; see SparkGLM, SparkLogisticModel, SparkProbitGLM

Kernels

• Added FeatureMapKernel, representing kernels which can be explicitly decomposed into feature mappings.
• Added Matern half integer kernel GenericMaternKernel[I]
• Added block(S: String*) method to block any hyper-parameters of kernels.

Data Pipes

• Added Data Pipes API for Apache Spark RDDs.

    val num = 20
    val numbers = sc.parallelize(1 to num)

    val convPipe = RDDPipe((n: Int) => n.toDouble)

    val sqPipe = RDDPipe((x: Double) => x*x)

    val sqrtPipe = RDDPipe((x: Double) => math.sqrt(x))

    val resultPipe = RDDPipe((r: RDD[Double]) => r.reduce(_+_).toInt)

    val netPipeline = convPipe > sqPipe > sqrtPipe > resultPipe
    netPipeline(numbers)

• Added UnivariateGaussianScaler class for gaussian scaling of univariate data.

I/O

• Added support for reading MATLAB .mat files in the MAT object.

Bug Fixes

• Correction to utils.getStats method used for calculating mean and variance of data sets consisting of DenseVector[Double].
• Correction to gradient calculation in RBF kernel family.

Improvements

• Speed up of kernel gradient computation, kernel and kernel gradient matrices with respect to the model hyper-parameters now calculated in a single pass through the data.

1.4.1-beta.9

Additions

• Added Expectation operator for RandomVariable implementations in the io.github.mandar2812.dynaml.probability package object. Usage example given below.
• SkewGaussian, ExtendedSkewGaussian: An breeze implementation of the SkewGaussian and extended Skew-Gaussian distributions respectively
• PushforwardMap, DifferentiableMap added: PushforwardMap enables creating new random variables with defined density from base random variables.

import io.github.mandar2812.dynaml.analysis._
import io.github.mandar2812.dynaml.probability._
import io.github.mandar2812.dynaml.probability.distributions._

val g = GaussianRV(0.0, 0.25)
val sg = RandomVariable(SkewGaussian(1.0, 0.0, 0.25))

//Define a determinant implementation for the Jacobian type (Double in this case)
implicit val detImpl = identityPipe[Double]

//Defines a homeomorphism y = exp(x) x = log(y)
val h: PushforwardMap[Double, Double, Double] = PushforwardMap(
  DataPipe((x: Double) => math.exp(x)),
  DifferentiableMap(
    (x: Double) => math.log(x),
    (x: Double) => 1.0/x)
)

//Creates a log-normal random variable
val p = h->g

//Creates a log-skew-gaussian random variable
val q = h->sg

//Calculate expectation of q
println("E[Q] = "+E(q))

• Added Markov Chain Monte Carlo (MCMC) based inference schemes ContinuousMCMC and the underlying sampling implementation in GeneralMetropolisHastings.
• Added implementation of Approximate Bayesian Computation (ABC) in the ApproxBayesComputation class.

Refactoring

1. Removed ProbabilityModel and replaced with JointProbabilityScheme and BayesJointProbabilityScheme, major refactoring to RandomVariable API.
2. Refactored GPMLOptimizer to GradBasedGlobalOptimizer

v1.4.1-beta.8

Fixes

1. minMaxScalingTrainTest minMaxScaling of DynaMLPipe using GaussianScaler instead of MinMaxScaler for processing of features.

v1.4.1-beta.7

Additions

1. DecomposableCovariance class now ready

DynaML>implicit val ev = VectorField(6) 
ev: VectorField = io.github.mandar2812.dynaml.analysis.VectorField@317221de
DynaML>implicit val sp = breezeDVSplitEncoder(2) 
sp: Encoder[DenseVector[Double], Array[DenseVector[Double]]] = io.github.mandar2812.dynaml.pipes.Encoder$$anon$2@75701980
DynaML>implicit val sumR = sumReducer  
sumR: DataPipe[Array[Double], Double] = io.github.mandar2812.dynaml.pipes.DataPipe$$anon$5@2ca540c9
DynaML>val kernel = new LaplacianKernel(1.5) 
kernel: LaplacianKernel = io.github.mandar2812.dynaml.kernels.LaplacianKernel@59e6d288
DynaML>val other_kernel = new PolynomialKernel(1, 0.05) 
other_kernel: PolynomialKernel = io.github.mandar2812.dynaml.kernels.PolynomialKernel@33fc593f
DynaML>other_kernel.block_all_hyper_parameters 
DynaML>val decompKernel = new DecomposableCovariance(kernel, other_kernel)(sp, sumReducer) 
decompKernel: DecomposableCovariance[DenseVector[Double]] = io.github.mandar2812.dynaml.kernels.DecomposableCovariance@7084fc88
DynaML>val other_kernel1 = new FBMKernel(1.0) 
other_kernel1: FBMKernel = io.github.mandar2812.dynaml.kernels.FBMKernel@350c07bb
DynaML>val decompKernel1 = new DecomposableCovariance(decompKernel, other_kernel1)(sp, sumReducer) 
decompKernel1: DecomposableCovariance[DenseVector[Double]] = io.github.mandar2812.dynaml.kernels.DecomposableCovariance@116d68cd
DynaML>decompKernel.hyper_parameters  
res10: List[String] = List("LaplacianKernel@59e6d288/beta", "PolynomialKernel@33fc593f/degree", "PolynomialKernel@33fc593f/offset")
DynaML>decompKernel.state 
res11: Map[String, Double] = Map("LaplacianKernel@59e6d288/beta" -> 1.5, "PolynomialKernel@33fc593f/degree" -> 1.0, "PolynomialKernel@33fc593f/offset" -> 0.05)
DynaML>decompKernel.blocked_hyper_parameters  
res12: List[String] = List("PolynomialKernel@33fc593f/degree", "PolynomialKernel@33fc593f/offset")
DynaML>decompKernel1.blocked_hyper_parameters  
res13: List[String] = List("DecomposableCovariance@7084fc88/PolynomialKernel@33fc593f/degree", "DecomposableCovariance@7084fc88/PolynomialKernel@33fc593f/offset")
DynaML>decompKernel1.state 
res14: Map[String, Double] = Map("DecomposableCovariance@7084fc88/LaplacianKernel@59e6d288/beta" -> 1.5, "DecomposableCovariance@7084fc88/PolynomialKernel@33fc593f/degree" -> 1.0, "DecomposableCovariance@7084fc88/PolynomialKernel@33fc593f/offset" -> 0.05, "FBMKernel@350c07bb/hurst" -> 1.0)
DynaML>decompKernel1.hyper_parameters  
res15: List[String] = List("DecomposableCovariance@7084fc88/LaplacianKernel@59e6d288/beta", "DecomposableCovariance@7084fc88/PolynomialKernel@33fc593f/degree", "DecomposableCovariance@7084fc88/PolynomialKernel@33fc593f/offset", "FBMKernel@350c07bb/hurst")
DynaML>val veca = DenseVector.tabulate[Double](8)(math.sin(_)) 
veca: DenseVector[Double] = DenseVector(0.0, 0.8414709848078965, 0.9092974268256817, 0.1411200080598672, -0.7568024953079282, -0.9589242746631385, -0.27941549819892586, 0.6569865987187891)
DynaML>val vecb = DenseVector.tabulate[Double](8)(math.cos(_)) 
vecb: DenseVector[Double] = DenseVector(1.0, 0.5403023058681398, -0.4161468365471424, -0.9899924966004454, -0.6536436208636119, 0.28366218546322625, 0.9601702866503661, 0.7539022543433046)
DynaML>decompKernel1.evaluate(veca, vecb) 
Dec 15, 2016 2:36:09 PM com.github.fommil.jni.JniLoader liberalLoad
INFO: successfully loaded /var/folders/5j/mhg359d57271g_hqg92jl5900000gn/T/jniloader4337296113710981207netlib-native_system-osx-x86_64.jnilib
res18: Double = -0.09808598779034527
DynaML>decompKernel1.evaluate(vecb, vecb) 
res19: Double = 2.153263332893377
DynaML>decompKernel1.evaluate(vecb, veca) 
res20: Double = -0.09808598779034527