Merge pull request #6 from rsepassi/master

v1.0.3

Author: lukaszkaiser

README.md

@@ -1,25 +1,45 @@
 # T2T: Tensor2Tensor Transformers
 
+[![PyPI
+version](https://badge.fury.io/py/tensor2tensor.svg)](https://badge.fury.io/py/tensor2tensor)
+[![GitHub
+Issues](https://img.shields.io/github/issues/tensorflow/tensor2tensor.svg)](https://github.com/tensorflow/tensor2tensor/issues)
+[![Contributions
+welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](CONTRIBUTING.md)
+[![License](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://opensource.org/licenses/Apache-2.0)
+
 [T2T](https://github.com/tensorflow/tensor2tensor) is a modular and extensible
-library and binaries for supervised learning with TensorFlow and with a focus on
-sequence tasks. Actively used and maintained by researchers and engineers within
-Google Brain, T2T strives to maximize idea bandwidth and minimize execution
-latency.
-
-T2T is particularly well-suited to researchers working on sequence tasks. We're
-eager to collaborate with you on extending T2T's powers, so please feel free to
-open an issue on GitHub to kick off a discussion and send along pull requests,
-See [our contribution doc](CONTRIBUTING.md) for details and our [open
+library and binaries for supervised learning with TensorFlow and with support
+for sequence tasks. It is actively used and maintained by researchers and
+engineers within the Google Brain team.
+
+We're eager to collaborate with you on extending T2T, so please feel
+free to [open an issue on
+GitHub](https://github.com/tensorflow/tensor2tensor/issues) or
+send along a pull request to add your data-set or model.
+See [our contribution
+doc](CONTRIBUTING.md) for details and our [open
 issues](https://github.com/tensorflow/tensor2tensor/issues).
 
-## T2T overview
+---
+
+## Walkthrough
+
+Here's a walkthrough training a good English-to-German translation
+model using the Transformer model from [*Attention Is All You
+Need*](https://arxiv.org/abs/1706.03762) on WMT data.
 
 ```
 pip install tensor2tensor
 
+# See what problems, models, and hyperparameter sets are available.
+# You can easily swap between them (and add new ones).
+t2t-trainer --registry_help
+
 PROBLEM=wmt_ende_tokens_32k
 MODEL=transformer
 HPARAMS=transformer_base
+
 DATA_DIR=$HOME/t2t_data
 TMP_DIR=/tmp/t2t_datagen
 TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS
@@ -35,6 +55,7 @@ t2t-datagen \
 mv $TMP_DIR/tokens.vocab.32768 $DATA_DIR
 
 # Train
+# *  If you run out of memory, add --hparams='batch_size=2048' or even 1024.
 t2t-trainer \
   --data_dir=$DATA_DIR \
   --problems=$PROBLEM \
@@ -59,23 +80,63 @@ t2t-trainer \
   --output_dir=$TRAIN_DIR \
   --train_steps=0 \
   --eval_steps=0 \
-  --beam_size=$BEAM_SIZE \
-  --alpha=$ALPHA \
+  --decode_beam_size=$BEAM_SIZE \
+  --decode_alpha=$ALPHA \
   --decode_from_file=$DECODE_FILE
 
 cat $DECODE_FILE.$MODEL.$HPARAMS.beam$BEAM_SIZE.alpha$ALPHA.decodes
 ```
 
-T2T modularizes training into several components, each of which can be seen in
-use in the above commands.
+---
 
-See the models, problems, and hyperparameter sets that are available:
+## Installation
 
-`t2t-trainer --registry_help`
+```
+pip install tensor2tensor
+```
+
+Binaries:
+
+```
+# Data generator
+t2t-datagen
+
+# Trainer
+t2t-trainer --registry_help
+```
+
+Library usage:
+
+```
+python -c "from tensor2tensor.models.transformer import Transformer"
+```
+
+---
+
+## Features
+
+* Many state of the art and baseline models are built-in and new models can be
+  added easily (open an issue or pull request!).
+* Many datasets across modalities - text, audio, image - available for
+  generation and use, and new ones can be added easily (open an issue or pull
+  request for public datasets!).
+* Models can be used with any dataset and input mode (or even multiple); all
+  modality-specific processing (e.g. embedding lookups for text tokens) is done
+  with `Modality` objects, which are specified per-feature in the dataset/task
+  specification.
+* Support for multi-GPU machines and synchronous (1 master, many workers) and
+  asynchrounous (independent workers synchronizing through a parameter server)
+  distributed training.
+* Easily swap amongst datasets and models by command-line flag with the data
+  generation script `t2t-datagen` and the training script `t2t-trainer`.
+
+---
+
+## T2T overview
 
 ### Datasets
 
-**Datasets** are all standardized on TFRecord files with `tensorflow.Example`
+**Datasets** are all standardized on `TFRecord` files with `tensorflow.Example`
 protocol buffers. All datasets are registered and generated with the
 [data
 generator](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/bin/t2t-datagen)
@@ -125,10 +186,12 @@ hyperparameters can be overriden with the `--hparams` flag. `--schedule` and
 related flags control local and distributed training/evaluation
 ([distributed training documentation](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/docs/distributed_training.md)).
 
+---
+
 ## Adding a dataset
 
-See the data generators
-[README](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/data_generators/README.md).
+See the [data generators
+README](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/data_generators/README.md).
 
 ---
 

setup.py

@@ -5,7 +5,7 @@
 
 setup(
     name='tensor2tensor',
-    version='1.0.2',
+    version='1.0.3',
     description='Tensor2Tensor',
     author='Google Inc.',
     author_email='[email protected]',

tensor2tensor.egg-info/PKG-INFO

@@ -0,0 +1,16 @@
+Metadata-Version: 1.1
+Name: tensor2tensor
+Version: 1.0.3
+Summary: Tensor2Tensor
+Home-page: http://github.com/tensorflow/tensor2tensor
+Author: Google Inc.
+Author-email: [email protected]
+License: Apache 2.0
+Description: UNKNOWN
+Keywords: tensorflow
+Platform: UNKNOWN
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence

tensor2tensor.egg-info/SOURCES.txt

@@ -0,0 +1,73 @@
+tensor2tensor/__init__.py
+tensor2tensor.egg-info/PKG-INFO
+tensor2tensor.egg-info/SOURCES.txt
+tensor2tensor.egg-info/dependency_links.txt
+tensor2tensor.egg-info/requires.txt
+tensor2tensor.egg-info/top_level.txt
+tensor2tensor/bin/t2t-datagen
+tensor2tensor/bin/t2t-trainer
+tensor2tensor/data_generators/__init__.py
+tensor2tensor/data_generators/algorithmic.py
+tensor2tensor/data_generators/algorithmic_math.py
+tensor2tensor/data_generators/algorithmic_math_test.py
+tensor2tensor/data_generators/algorithmic_test.py
+tensor2tensor/data_generators/audio.py
+tensor2tensor/data_generators/audio_test.py
+tensor2tensor/data_generators/concatenate_examples.py
+tensor2tensor/data_generators/generator_utils.py
+tensor2tensor/data_generators/generator_utils_test.py
+tensor2tensor/data_generators/image.py
+tensor2tensor/data_generators/image_test.py
+tensor2tensor/data_generators/lm_example.py
+tensor2tensor/data_generators/problem_hparams.py
+tensor2tensor/data_generators/problem_hparams_test.py
+tensor2tensor/data_generators/replace_oov.py
+tensor2tensor/data_generators/snli.py
+tensor2tensor/data_generators/text_encoder.py
+tensor2tensor/data_generators/text_encoder_build_subword.py
+tensor2tensor/data_generators/text_encoder_inspect_subword.py
+tensor2tensor/data_generators/tokenizer.py
+tensor2tensor/data_generators/tokenizer_test.py
+tensor2tensor/data_generators/wmt.py
+tensor2tensor/data_generators/wmt_test.py
+tensor2tensor/data_generators/wsj_parsing.py
+tensor2tensor/models/__init__.py
+tensor2tensor/models/attention_lm.py
+tensor2tensor/models/attention_lm_moe.py
+tensor2tensor/models/bytenet.py
+tensor2tensor/models/bytenet_test.py
+tensor2tensor/models/common_attention.py
+tensor2tensor/models/common_hparams.py
+tensor2tensor/models/common_layers.py
+tensor2tensor/models/common_layers_test.py
+tensor2tensor/models/lstm.py
+tensor2tensor/models/lstm_test.py
+tensor2tensor/models/models.py
+tensor2tensor/models/multimodel.py
+tensor2tensor/models/multimodel_test.py
+tensor2tensor/models/neural_gpu.py
+tensor2tensor/models/neural_gpu_test.py
+tensor2tensor/models/slicenet.py
+tensor2tensor/models/slicenet_test.py
+tensor2tensor/models/transformer.py
+tensor2tensor/models/transformer_test.py
+tensor2tensor/models/xception.py
+tensor2tensor/models/xception_test.py
+tensor2tensor/utils/__init__.py
+tensor2tensor/utils/avg_checkpoints.py
+tensor2tensor/utils/beam_search.py
+tensor2tensor/utils/beam_search_test.py
+tensor2tensor/utils/bleu_hook.py
+tensor2tensor/utils/bleu_hook_test.py
+tensor2tensor/utils/data_reader.py
+tensor2tensor/utils/data_reader_test.py
+tensor2tensor/utils/expert_utils.py
+tensor2tensor/utils/metrics.py
+tensor2tensor/utils/metrics_test.py
+tensor2tensor/utils/modality.py
+tensor2tensor/utils/modality_test.py
+tensor2tensor/utils/registry.py
+tensor2tensor/utils/registry_test.py
+tensor2tensor/utils/t2t_model.py
+tensor2tensor/utils/trainer_utils.py
+tensor2tensor/utils/trainer_utils_test.py

tensor2tensor.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

tensor2tensor.egg-info/requires.txt

@@ -0,0 +1,4 @@
+numpy
+sympy
+six
+tensorflow-gpu>=1.2.0rc1

tensor2tensor.egg-info/top_level.txt

@@ -0,0 +1 @@
+tensor2tensor

tensor2tensor/bin/make_tf_configs.py

@@ -32,6 +32,7 @@
 
 # Dependency imports
 
+import six
 import tensorflow as tf
 
 flags = tf.flags
@@ -50,7 +51,7 @@ def main(_):
 
   cluster = {"ps": ps, "worker": workers}
 
-  for task_type, jobs in [("worker", workers), ("ps", ps)]:
+  for task_type, jobs in six.iteritems(cluster):
     for idx, job in enumerate(jobs):
       if task_type == "worker":
         cmd_line_flags = " ".join([

tensor2tensor/data_generators/algorithmic.py

@@ -28,7 +28,7 @@ def identity_generator(nbr_symbols, max_length, nbr_cases):
   """Generator for the identity (copy) task on sequences of symbols.
 
   The length of the sequence is drawn uniformly at random from [1, max_length]
-  and then symbols are drawn uniformly at random from [1, nbr_symbols] until
+  and then symbols are drawn uniformly at random from [2, nbr_symbols] until
   nbr_cases sequences have been produced.
 
   Args:
@@ -42,15 +42,15 @@ def identity_generator(nbr_symbols, max_length, nbr_cases):
   """
   for _ in xrange(nbr_cases):
     l = np.random.randint(max_length) + 1
-    inputs = [np.random.randint(nbr_symbols) + 1 for _ in xrange(l)]
+    inputs = [np.random.randint(nbr_symbols) + 2 for _ in xrange(l)]
     yield {"inputs": inputs, "targets": inputs}
 
 
 def shift_generator(nbr_symbols, shift, max_length, nbr_cases):
   """Generator for the shift task on sequences of symbols.
 
   The length of the sequence is drawn uniformly at random from [1, max_length]
-  and then symbols are drawn uniformly at random from [1, nbr_symbols - shift]
+  and then symbols are drawn uniformly at random from [2, nbr_symbols - shift]
   until nbr_cases sequences have been produced (output[i] = input[i] + shift).
 
   Args:
@@ -65,15 +65,15 @@ def shift_generator(nbr_symbols, shift, max_length, nbr_cases):
   """
   for _ in xrange(nbr_cases):
     l = np.random.randint(max_length) + 1
-    inputs = [np.random.randint(nbr_symbols - shift) + 1 for _ in xrange(l)]
+    inputs = [np.random.randint(nbr_symbols - shift) + 2 for _ in xrange(l)]
     yield {"inputs": inputs, "targets": [i + shift for i in inputs]}
 
 
 def reverse_generator(nbr_symbols, max_length, nbr_cases):
   """Generator for the reversing task on sequences of symbols.
 
   The length of the sequence is drawn uniformly at random from [1, max_length]
-  and then symbols are drawn uniformly at random from [1, nbr_symbols] until
+  and then symbols are drawn uniformly at random from [2, nbr_symbols] until
   nbr_cases sequences have been produced.
 
   Args:
@@ -87,7 +87,7 @@ def reverse_generator(nbr_symbols, max_length, nbr_cases):
   """
   for _ in xrange(nbr_cases):
     l = np.random.randint(max_length) + 1
-    inputs = [np.random.randint(nbr_symbols) + 1 for _ in xrange(l)]
+    inputs = [np.random.randint(nbr_symbols) + 2 for _ in xrange(l)]
     yield {"inputs": inputs, "targets": list(reversed(inputs))}
 
 
@@ -139,8 +139,8 @@ def addition_generator(base, max_length, nbr_cases):
     n2 = random_number_lower_endian(l2, base)
     result = lower_endian_to_number(n1, base) + lower_endian_to_number(n2, base)
     # We shift digits by 1 on input and output to leave 0 for padding.
-    inputs = [i + 1 for i in n1] + [base + 1] + [i + 1 for i in n2]
-    targets = [i + 1 for i in number_to_lower_endian(result, base)]
+    inputs = [i + 2 for i in n1] + [base + 2] + [i + 2 for i in n2]
+    targets = [i + 2 for i in number_to_lower_endian(result, base)]
     yield {"inputs": inputs, "targets": targets}
 
 
@@ -173,6 +173,6 @@ def multiplication_generator(base, max_length, nbr_cases):
     n2 = random_number_lower_endian(l2, base)
     result = lower_endian_to_number(n1, base) * lower_endian_to_number(n2, base)
     # We shift digits by 1 on input and output to leave 0 for padding.
-    inputs = [i + 1 for i in n1] + [base + 1] + [i + 1 for i in n2]
-    targets = [i + 1 for i in number_to_lower_endian(result, base)]
+    inputs = [i + 2 for i in n1] + [base + 2] + [i + 2 for i in n2]
+    targets = [i + 2 for i in number_to_lower_endian(result, base)]
     yield {"inputs": inputs, "targets": targets}