Prepare for initial 0.1.0 release

Author: kevin-bates

CONTRIBUTING.rst

@@ -1,5 +1,3 @@
-.. highlight:: shell
-
 ============
 Contributing
 ============
@@ -76,14 +74,12 @@ Ready to contribute? Here's how to set up `yarn_kernel_provider` for local devel
 
    Now you can make your changes locally.
 
-5. When you're done making changes, check that your changes pass flake8 and the
-   tests, including testing other Python versions with tox::
+5. When you're done making changes, check that your changes pass flake8 and the tests::
 
-    $ flake8 yarn_kernel_provider tests
-    $ python setup.py test or py.test
-    $ tox
+    $ flake8 yarn_kernel_provider
+    $ make test or py.test
 
-   To get flake8 and tox, just pip install them into your virtualenv.
+   To get flake8, just pip install it into your virtualenv.
 
 6. Commit your changes and push your branch to GitHub::
 
@@ -102,17 +98,10 @@ Before you submit a pull request, check that it meets these guidelines:
 2. If the pull request adds functionality, the docs should be updated. Put
    your new functionality into a function with a docstring, and add the
    feature to the list in README.rst.
-3. The pull request should work for Python 2.7, 3.4, 3.5 and 3.6, and for PyPy. Check
+3. The pull request should work for Python 3.4, 3.5 and 3.6, and for PyPy. Check
    https://travis-ci.org/gateway-experiments/yarn_kernel_provider/pull_requests
    and make sure that the tests pass for all supported Python versions.
 
-Tips
-----
-
-To run a subset of tests::
-
-
-    $ python -m unittest tests.test_yarn_kernel_provider
 
 Deploying
 ---------

HISTORY.rst

@@ -2,7 +2,7 @@
 History
 =======
 
-0.1.0 (2019-05-30)
-------------------
+0.1.0 (Release date pending)
+----------------------------
 
 * First release on PyPI.

Makefile

@@ -64,7 +64,7 @@ coverage: ## check code coverage quickly with the default Python
 docs: ## generate Sphinx HTML documentation, including API docs
 	rm -f docs/yarn_kernel_provider.rst
 	rm -f docs/modules.rst
-	sphinx-apidoc -o docs/ yarn_kernel_provider
+	#sphinx-apidoc -o docs/ yarn_kernel_provider
 	$(MAKE) -C docs clean
 	$(MAKE) -C docs html
 	$(BROWSER) docs/_build/html/index.html

README.md

@@ -2,16 +2,109 @@
 
 __NOTE: This repository is experimental and undergoing frequent changes!__
 
-The Yarn Kernel Provider package provides support necessary for launching Jupyter kernels within YARN clusters.  This is accomplished via two classes:
+The Yarn Kernel Provider package provides support necessary for launching Jupyter kernels within YARN clusters.  It adheres to requirements set forth in the [Jupyter Kernel Management](https://github.com/takluyver/jupyter_kernel_mgmt) refactoring for kernel management and discovery. This is accomplished via two classes:
 
 1. [`YarnKernelProvider`](https://github.com/gateway-experiments/yarn_kernel_provider/blob/master/yarn_kernel_provider/provider.py) is invoked by the application to locate and identify specific kernel specificiations (kernelspecs) that manage kernel lifecycles within a YARN cluster.
 2. [`YarnKernelLifecycleManager`](https://github.com/gateway-experiments/yarn_kernel_provider/blob/master/yarn_kernel_provider/yarn.py) is instantiated by the [`RemoteKernelManager`](https://github.com/gateway-experiments/remote_kernel_provider/blob/master/remote_kernel_provider/manager.py) to peform the kernel lifecycle management.  It performs post-launch discovery of the application and handles its termination via the [YARN REST API](https://github.com/toidi/hadoop-yarn-api-python-client).
 
+Installation of yarn_kernel_provider also includes a Jupyter application that can be used to create appropriate kernel specifications relative to YARN Spark and Dask.
+
 ## Installation
-`YarnKernelProvider` is a pip-installable package:
+Yarn Kernel Provider is a pip-installable package:
 ```bash
 pip install yarn_kernel_provider
 ```
 
-## YARN Kernel Specifications
-Criteria for discovery of the kernel specification via the `YarnKernelProvider` is that a `kernel.json` file exist in a sub-directory of `yarn_kernels`.  
+##Usage
+Because this version of Jupyter kernel management is still in its experimental stages, a [special branch of Notebook](https://github.com/takluyver/notebook/tree/jupyter-kernel-mgmt) is required, which includes the machinery to leverage the new framework.  An installable build of this branch is available as an asset on the [interim-dev release](https://github.com/gateway-experiments/remote_kernel_provider/releases/tag/v0.1-interim-dev) of the Remote Kernel Provider on which Yarn Kernel Provider depends.
+
+### YARN Kernel Specifications
+Criteria for discovery of the kernel specification via the `YarnKernelProvider` is that a `yarnkp_kernel.json` file exist in a sub-directory named `kernels` in the Jupyter path hierarchy. 
+
+Such kernel specifications should be initially created using the included Jupyter application`jupyter-yarn-kernelspec` to insure the minimally viable requirements exist.  This application can be used to create specifications for YARN Spark and Dask.  Spark support is available for three languages: Python, Scala and R, while Dask support is available for Python.
+
+To create kernel specifications for use by YarnKernelProvider use `juptyer yarn-kernelspec install`.  Here are it's parameter options, produced using `jupyter yarn-kernelspec install --help`.  All parameters are optional with no parameters yielding a Python-based kernelspec for Spark on the local YARN cluster.  However, locations for SPARK_HOME and Python runtimes may likely require changes if not provided.
+
+```
+A Jupyter kernel for talking to Spark/Dask within a YARN cluster
+
+Options
+-------
+
+Arguments that take values are actually convenience aliases to full
+Configurables, whose aliases are listed on the help line. For more information
+on full configurables, see '--help-all'.
+
+--user
+    Install to the per-user kernel registry
+--sys-prefix
+    Install to Python's sys.prefix. Useful in conda/virtual environments.
+--dask
+    Install kernelspec for Dask YARN.
+--debug
+    set log level to logging.DEBUG (maximize logging output)
+--prefix=<Unicode> (YKP_SpecInstaller.prefix)
+    Default: ''
+    Specify a prefix to install to, e.g. an env. The kernelspec will be
+    installed in PREFIX/share/jupyter/kernels/
+--kernel_name=<Unicode> (YKP_SpecInstaller.kernel_name)
+    Default: 'yarnkp_spark_python'
+    Install the kernel spec into a directory with this name.
+--display_name=<Unicode> (YKP_SpecInstaller.display_name)
+    Default: 'Spark Python (YARN Cluster)'
+    The display name of the kernel - used by user-facing applications.
+--yarn_endpoint=<Unicode> (YKP_SpecInstaller.yarn_endpoint)
+    Default: None
+    The http url specifying the YARN Resource Manager. Note: If this value is
+    NOT set, the YARN library will use the files within the local
+    HADOOP_CONFIG_DIR to determine the active resource manager.
+    (YKP_YARN_ENDPOINT env var)
+--alt_yarn_endpoint=<Unicode> (YKP_SpecInstaller.alt_yarn_endpoint)
+    Default: None
+    The http url specifying the alternate YARN Resource Manager.  This value
+    should be set when YARN Resource Managers are configured for high
+    availability.  Note: If both YARN endpoints are NOT set, the YARN library
+    will use the files within the local HADOOP_CONFIG_DIR to determine the
+    active resource manager. (YKP_ALT_YARN_ENDPOINT env var)
+--yarn_endpoint_security_enabled=<Bool> (YKP_SpecInstaller.yarn_endpoint_security_enabled)
+    Default: False
+    Is YARN Kerberos/SPNEGO Security enabled (True/False).
+    (YKP_YARN_ENDPOINT_SECURITY_ENABLED env var)
+--language=<Unicode> (YKP_SpecInstaller.language)
+    Default: 'Python'
+    The language of the underlying kernel.  Must be one of 'Python', 'R', or
+    'Scala'.  Default = 'Python'.
+--python_root=<Unicode> (YKP_SpecInstaller.python_root)
+    Default: '/opt/conda'
+    Specify where the root of the python installation resides (parent dir of
+    bin/python).
+--spark_home=<Unicode> (YKP_SpecInstaller.spark_home)
+    Default: '/usr/hdp/current/spark2-client'
+    Specify where the spark files can be found.
+--spark_init_mode=<Unicode> (YKP_SpecInstaller.spark_init_mode)
+    Default: 'lazy'
+    Spark context initialization mode.  Must be one of 'lazy', 'eager', or
+    'none'.  Default = 'lazy'.
+--extra_spark_opts=<Unicode> (YKP_SpecInstaller.extra_spark_opts)
+    Default: ''
+    Specify additional Spark options.
+--extra_dask_opts=<Unicode> (YKP_SpecInstaller.extra_dask_opts)
+    Default: ''
+    Specify additional Dask options.
+--log-level=<Enum> (Application.log_level)
+    Default: 30
+    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
+    Set the log level by value or name.
+--config=<Unicode> (JupyterApp.config_file)
+    Default: ''
+    Full path of a config file.
+
+To see all available configurables, use `--help-all`
+
+Examples
+--------
+
+    jupyter-yarn-kernelspec install --language=R --spark_home=/usr/local/spark
+    jupyter-yarn-kernelspec install --kernel_name=dask_python --dask --yarn_endpoint=http://foo.bar:8088/ws/v1/cluster
+    jupyter-yarn-kernelspec install --language=Scala --spark_init_mode='eager'
+``` 

docs/conf.py

@@ -32,7 +32,7 @@
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode']
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'm2r']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -41,7 +41,7 @@
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
 
 # The master toctree document.
 master_doc = 'index'

docs/index.rst

@@ -1,5 +1,5 @@
 Welcome to Yarn Kernel Provider's documentation!
-======================================
+================================================
 
 .. toctree::
    :maxdepth: 2
@@ -8,7 +8,6 @@ Welcome to Yarn Kernel Provider's documentation!
    readme
    installation
    usage
-   modules
    contributing
    history
 

docs/readme.rst

@@ -1 +1 @@
-.. include:: ../README.rst
+.. mdinclude:: ../README.md

requirements_dev.txt

@@ -1,11 +1,11 @@
-pip==18.1
-bumpversion==0.5.3
-wheel==0.32.1
-watchdog==0.9.0
-flake8==3.5.0
-tox==3.5.2
-coverage==4.5.1
-Sphinx==1.8.1
-twine==1.12.1
+pip>=18.1
+bumpversion>=0.5.3
+wheel>=0.32.1
+watchdog>=0.9.0
+flake8>=3.5.0
+tox>=3.5.2
+coverage>=4.5.1
+Sphinx>=1.8.1
+twine>=1.12.1
 
 

tox.ini

@@ -1,12 +1,11 @@
 [tox]
-envlist = py27, py34, py35, py36, flake8
+envlist = py34, py35, py36, flake8
 
 [travis]
 python =
     3.6: py36
     3.5: py35
     3.4: py34
-    2.7: py27
 
 [testenv:flake8]
 basepython = python
@@ -17,5 +16,5 @@ commands = flake8 yarn_kernel_provider
 setenv =
     PYTHONPATH = {toxinidir}
 
-commands = python setup.py test
+commands = make test
 

yarn_kernel_provider/kernelspecapp.py

@@ -43,11 +43,11 @@
 class YKP_SpecInstaller(JupyterApp):
     """CLI for extension management."""
     name = u'jupyter-yarn-kernelspec'
-    description = u'A Jupyter kernel for talking to Spark within a YARN cluster'
+    description = u'A Jupyter kernel for talking to Spark/Dask within a YARN cluster'
     examples = '''
     jupyter-yarn-kernelspec install --language=R --spark_home=/usr/local/spark
     jupyter-yarn-kernelspec install --kernel_name=dask_python --dask --yarn_endpoint=http://foo.bar:8088/ws/v1/cluster
-    jupyter-yarn-kernelspec install --spark_init_mode='eager'
+    jupyter-yarn-kernelspec install --language=Scala --spark_init_mode='eager'
     '''
     kernel_spec_manager = Instance(KernelSpecManager)
 
@@ -282,17 +282,17 @@ def _log_and_exit(self, msg, exit_status=1):
 
 class YarnKernelProviderApp(Application):
     version = __version__
-    name = 'jupyter yarn-kernel-provider'
+    name = 'jupyter yarn-kernelspec'
     description = '''Application used to create kernelspecs for use on Hadoop YARN clusters
 
     \tYarn Kernel Provider Version: {}
     '''.format(__version__)
     examples = '''
-    jupyter yarn_kernel_provider install-spec - Installs the kernel as a Jupyter Kernel.
+    jupyter yarn-kernelspec install - Installs the kernel as a Jupyter Kernel.
     '''
 
     subcommands = Dict({
-        'install-spec': (YKP_SpecInstaller, YKP_SpecInstaller.description.splitlines()[0]),
+        'install': (YKP_SpecInstaller, YKP_SpecInstaller.description.splitlines()[0]),
     })
 
     aliases = {}