Spring of Rebirth

.gitignore

@@ -0,0 +1,9 @@
+node_modules
+
+datasets/**/*.npz
+*.hdf5
+
+yarn-error.log
+.DS_STORE
+
+.pyc

.prettierrc

@@ -0,0 +1,9 @@
+{
+  "bracketSpacing": true,
+  "printWidth": 120,
+  "semi": true,
+  "singleQuote": true,
+  "tabWidth": 2,
+  "trailingComma": "all",
+  "useTabs": false
+}

.vscode/settings.json

@@ -0,0 +1,8 @@
+{
+  "cSpell.words": [
+    "ncells",
+    "numpy",
+    "scipy",
+    "sklearn"
+  ]
+}

README.md

@@ -1,38 +1,35 @@
-### Installing Python libraries
-
-To run SPRING Viewer locally, make sure Python 2.7 is installed (and that it's your active version). You will also need the following Python libraries:  
-
-`scikit-learn`  
-`numpy`  
-`scipy`  
-`matplotlib`  
-`h5py`  
-`networkx`  
-`fa2`  
-`python-louvain`
+# SPRING
 
-We recommend Anaconda to manage your Python libraries. You can download it here (be sure to get the Python 2.7 version):   https://conda.io/miniconda.html. Libraries can then be installed using the command `conda`. To do so, open Terminal (Mac) or Anaconda Prompt (Windows) and enter:  
+<!-- TOC -->
 
-`conda install scikit-learn numpy scipy matplotlib h5py`
+- [SPRING](#spring)
+  - [Setting Up A SPRING Data Directory](#setting-up-a-spring-data-directory)
+  - [Backend](#backend)
+    - [Installing Python libraries](#installing-python-libraries)
+  - [Frontend](#frontend)
+    - [Development Build](#development-build)
+    - [Production Build](#production-build)
+    - [RequireJS](#requirejs)
+  - [Running SPRING Viewer](#running-spring-viewer)
 
-The remaining libraries can be installed using `pip`. Note that if you're a Windows user, you'll first need to install Microsoft Visual C++ compiler for Python (available from http://aka.ms/vcpython27). Enter the following into Terminal or Anaconda Prompt:  
+<!-- /TOC -->
 
-`pip install networkx fa2 python-louvain`
+[SPRING](https://doi.org/10.1093/bioinformatics/btx792) is a collection of pre-processing scripts and a web browser-based tool for visualizing and interacting with high dimensional data.
 
+## Setting Up A SPRING Data Directory
 
-### Setting up a SPRING data directory
 See the example notebooks:  
 [Hematopoietic progenitor FACS subpopulations](./data_prep/spring_example_HPCs.ipynb)  
-[Mature blood cells (10X Genomics 4k PBMCs)](./data_prep/spring_example_pbmc4k.ipynb)  
+[Mature blood cells (10X Genomics 4k PBMCs)](./data_prep/spring_example_pbmc4k.ipynb)
 
-A SPRING data set consist of a main directory and any number of subdirectories, with each subdirectory corresponding to one SPRING plot (i.e. subplot) that draws on a data matrix stored in the main directory. The main directory should have the following files, as well as one subdirectory for each SPRING plot. 
+A SPRING data set consist of a main directory and any number of subdirectories, with each subdirectory corresponding to one SPRING plot (i.e. subplot) that draws on a data matrix stored in the main directory. The main directory should have the following files, as well as one subdirectory for each SPRING plot.
 
 `counts_norm.npz`  
 `counts_norm_sparse_cells.hdf5`  
 `counts_norm_sparse_genes.hdf5`  
-`genes.txt`  
+`genes.txt`
 
-Each subdirectory should contain:  
+Each subdirectory should contain:
 
 `categorical_coloring_data.json`  
 `cell_filter.npy`  
@@ -42,15 +39,65 @@ Each subdirectory should contain:
 `coordinates.txt`  
 `edges.csv`  
 `graph_data.json`  
-`run_info.json`  
+`run_info.json`
+
+Place the main directory somewhere inside folder that contains this README and the other SPRING file. We recommend that you create a special `datasets` directory. For example, if you have a main data set called `human_bone_marrow` and another called `frog_embryo`, you could place them in `./datasets/human_bone_marrow/` and `./datasets/frog_embryo/`.
+
+## Backend
+
+### Installing Python libraries
+
+To run SPRING Viewer locally, make sure Python 2.7 is installed (and that it's your active version). You will also need the following Python libraries:
+
+`scikit-learn`  
+`numpy`  
+`scipy`  
+`h5py`  
+`networkx`  
+`fa2`  
+`python-louvain`
+
+We recommend Anaconda to manage your Python libraries. You can download it here (be sure to get the Python 2.7 version): https://conda.io/miniconda.html. Libraries can then be installed using the command `conda`. To do so, open Terminal (Mac) or Anaconda Prompt (Windows) and enter:
+
+`conda install scikit-learn numpy scipy h5py`
+
+The remaining libraries can be installed using `pip`. Note that if you're a Windows user, you'll first need to install Microsoft Visual C++ compiler for Python (available from http://aka.ms/vcpython27). Enter the following into Terminal or Anaconda Prompt:
+
+`pip install networkx fa2 python-louvain`
+
+## Frontend
 
-Place the main directory somehwere inside folder that contains this README and the other SPRING file. We recommend that you create a special `datasets` directory. For example, if you have a main data set called `human_bone_marrow` and another called `frog_embryo`, you could place them in `./datasets/human_bone_marrow/` and `./datasets/frog_embryo/`. 
+The SPRING frontend is setup as a JavaScript module using [Yarn](https://yarnpkg.com/en/) as a package manager and [TypeScript](https://www.typescriptlang.org/) as a transpiler - Meaning we can write code that uses features like [async/await](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function) while still supporting older browsers!
 
+### Development Build
 
-### Running SPRING Viewer
+To get the frontend building for development purpose, open your favorite terminal/shell into this directory and then run:
 
-1. Open Terminal (Mac) or Anaconda Prompt (Windows) and change directories (`cd`) to the directory containing this README file (`SPRING_dev/`). 
+```sh
+yarn
+yarn build --watch
+```
+
+This will compile code inside `src/` and put it inside `dist/` as well as watch for file changes and re-compile as needed. Neat!
+
+### Production Build
+
+Similar to the above, run the following two commands to create the production build:
+
+```sh
+yarn
+yarn build
+```
+
+### RequireJS
+
+Further, please note there is only a single entry point for the app on the springViewer.html page - namely, `app.js`. Digging down a bit, you'll see that it is setup as a [RequireJS](https://requirejs.org/) module and is **pulling code from the vendor and dist directories**!
+
+This setup seems to provide the most flexibility in allowing spring to come bundled with minified 3rd party library code (d3, jquery, pixi.js, etc), while still allowing the app code to be setup in a modular way and not adding in a lot of extra configuration using tools like Rollup or Webpack.
+
+## Running SPRING Viewer
+
+1. Open Terminal (Mac) or Anaconda Prompt (Windows) and change directories (`cd`) to the directory containing this README file (`SPRING_dev/`).
 2. Start a local server by entering the following: `python -m CGIHTTPServer 8000`
 3. Open web browser (preferably Chrome; best to use incognito mode to ensure no cached data is used).
-4. View data set by navigating to corresponding URL: http://localhost:8000/springViewer_1_6_dev.html?path_to/main/subplot. In the example above, if you wanted to view a SPRING plot called `HSC` in the main directory `human_bone_marrow`, then you would navigate to http://localhost:8000/springViewer_1_6_dev.html?datasets/human_bone_marrow/HSC
-
+4. View data set by navigating to corresponding URL: http://localhost:8000/springViewer.html?path_to/main/subplot. In the example above, if you wanted to view a SPRING plot called `HSC` in the main directory `human_bone_marrow`, then you would navigate to http://localhost:8000/springViewer.html?datasets/human_bone_marrow/HSC

app.js

@@ -0,0 +1,57 @@
+requirejs.config({
+  baseUrl: 'vendor',
+  shim: {
+    PIXI: { exports: 'PIXI' },
+  },
+  paths: {
+    dist: '../dist',
+    d3: 'd3.v5.min',
+    html2canvas: 'html2canvas.min',
+    spinner: 'spin.min',
+    sweetalert: 'sweetalert.min',
+  },
+});
+
+if (!window.cacheData) {
+  window.cacheData = new Map();
+}
+
+window.addEventListener('message', event => {
+  if (!event.isTrusted && event.origin === window.location.origin) {
+    return;
+  }
+  try {
+    if (typeof event.data === 'string') {
+      const parsedData = JSON.parse(event.data);
+      switch (parsedData.type) {
+        case 'init': {
+          if (parsedData.payload.indices) {
+            window.cacheData.set('indices', parsedData.payload.indices);
+          }
+
+          if (parsedData.payload.categories) {
+            window.cacheData.set('categories', parsedData.payload.categories);
+          }
+        }
+        case 'selected-cells-update': {
+          if (parsedData.payload.coordinates) {
+            window.cacheData.set('selected-cells', parsedData.payload.coordinates);
+          }
+        }
+        default: {
+          break;
+        }
+      }
+    }
+  } catch (err) {
+    console.log(`Unable to parse received message.\n\
+    Data: ${event.data}
+    Error: ${err}`);
+  }
+});
+
+requirejs.config({
+  waitSeconds: 200,
+});
+
+requirejs(['dist/main']);

cgi-bin/apply_gene_set_retrospective.py

@@ -1,6 +1,9 @@
 #!/usr/bin/env python
 #========================================================================================#
-import numpy as np, sys, h5py, json
+import numpy as np
+import sys
+import h5py
+import json
 
 base_dir = sys.argv[1]
 sub_dirs = sys.argv[2]
@@ -9,58 +12,58 @@
 hf = h5py.File(base_dir + '/counts_norm_sparse_genes.hdf5', 'r')
 ncells = hf.attrs['ncells']
 valid_genes = hf.get('counts').keys()
-gene_map = {g.split()[0]:g for g in valid_genes}
+gene_map = {g.split()[0]: g for g in valid_genes}
 
 
 # Load gene sets
 gene_sets = {}
 all_genes = set([])
-for l in open(gene_sets_path).read().replace('\r','\n').split('\n'):
-	l = l.split('\t')
-	if len(l) > 1:
-		gene = l[0]
-		name = l[1]
-		if gene in gene_map: gene = gene_map[gene]
-		if not gene in valid_genes: print 'Invalid',gene
-		else:	
-			if not name in gene_sets:
-				gene_sets[name] = []
-			gene_sets[name].append(gene)
-			all_genes.add(gene)
+for l in open(gene_sets_path).read().replace('\r', '\n').split('\n'):
+    split_l = l.split('\t')
+    if len(l) > 1:
+        gene = split_l[0]
+        name = split_l[1]
+        if gene in gene_map:
+            gene = gene_map[gene]
+        if gene not in valid_genes:
+            print('Invalid', gene)
+        else:
+            if name not in gene_sets:
+                gene_sets[name] = []
+            gene_sets[name].append(gene)
+            all_genes.add(gene)
 
 
-# Load gene expression 
+# Load gene expression
 gene_exp = {}
 for g in all_genes:
-	ee = np.zeros(ncells)
-	counts = np.array(hf.get('counts').get(g))
-	cell_ix = np.array(hf.get('cell_ix').get(g))
-	ee[cell_ix] = counts
-	gene_exp[g] = ee
+    ee = np.zeros(ncells)
+    counts = np.array(hf.get('counts').get(g))
+    cell_ix = np.array(hf.get('cell_ix').get(g))
+    ee[cell_ix] = counts
+    gene_exp[g] = ee
 
 # compute scores
 scores = {}
-for k,gs in gene_sets.items():
-	Z = np.array([gene_exp[g] for g in gs])
-	Z = (Z - np.mean(Z,axis=1)[:,None]) / (np.std(Z,axis=1)[:,None] + .0001)
-	ss = np.sum(Z,axis=0)
-	ss = ss - np.min(ss)
-	scores[k] = ss
+for k, gs in gene_sets.items():
+    Z = np.array([gene_exp[g] for g in gs])
+    Z = (Z - np.mean(Z, axis=1)[:, None]) / \
+        (np.std(Z, axis=1)[:, None] + .0001)
+    ss = np.sum(Z, axis=0)
+    ss = ss - np.min(ss)
+    scores[k] = ss
 
 # Apply to each subplots
 for dd in sub_dirs.split(','):
-	cell_ix = np.load(base_dir+'/'+dd+'/cell_filter.npy')
-	f = open(base_dir+'/'+dd+'/color_data_gene_sets.csv','a')
-	for k,ss in scores.items():
-		newline = ','.join([k]+[repr(x) for x in ss[cell_ix]])
-		f.write(newline+'\n')
-	f.close()
-
-	color_stats = json.load(open(base_dir+'/'+dd+'/color_stats.json'))
-	for k,ss in scores.items(): 
-		color_stats[k] = (np.mean(ss),np.std(ss),np.min(ss),np.max(ss),np.percentile(ss,99))
-	json.dump(color_stats,open(base_dir+'/'+dd+'/color_stats.json','w'))
-
-
-
+    cell_ix = np.load(base_dir + '/' + dd + '/cell_filter.npy')
+    f = open(base_dir + '/' + dd + '/color_data_gene_sets.csv', 'a')
+    for k, ss in scores.items():
+        newline = ','.join([k] + [repr(x) for x in ss[cell_ix]])
+        f.write(newline + '\n')
+    f.close()
 
+    color_stats = json.load(open(base_dir + '/' + dd + '/color_stats.json'))
+    for k, ss in scores.items():
+        color_stats[k] = (np.mean(ss), np.std(ss), np.min(ss),
+                          np.max(ss), np.percentile(ss, 99))
+    json.dump(color_stats, open(base_dir + '/' + dd + '/color_stats.json', 'w'))