Merge pull request #685 from visualsilicon/visualsilicon/ml-agents

Visualsilicon/ml agents

Author: pareenaverma

.gitignore

@@ -8,3 +8,4 @@ package-lock.json
 
 # macOS files
 *.DS_Store
+nohup.out

content/learning-paths/smartphones-and-mobile/using_unity_machine_learning_agents/02-ML-in-games.md

@@ -0,0 +1,37 @@
+---
+title: Machine Learning in games
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+### What problem are we trying to solve?
+
+Machine Learning has many potential applications across various platforms. One particular use case is implementing realistic behavior for non-player characters and opponents in video games. In this learning path, we will utilize the Unity Machine Learning Agents toolkit to build an "AI brain" within a game environment.
+
+Creating rich, realistic AI for games is challenging, especially as project complexity increases. Traditional solutions often require extensive specialized coding and tools. With machine learning, we can avoid specialized code and instead leverage a generic approach to building smart AI-powered opponents with varying levels of difficulty.
+
+### What are we going to create?
+
+We will be using a Unity game template that we have prepared for this learning path to implement AI for a fighting game with two characters. The characters in this game will be battling in a 3D arena. One fighter will be controlled by the human player, while the other will be controlled by the AI brain we create.
+
+![ML Gameplay](images/ml-gameplay.png "Figure 1. Our fighting game with AI opponent")
+
+### How do this game template and its AI work?
+
+Machine learning applications are built in two stages:
+
+First, a _model_ (neural network) is trained in the appropriately-named _training_ stage.
+
+Next, this trained model is used at runtime in the final application (the game) to make smart decisions and actions. This is the _inference_ stage.
+
+Inference usually runs on a different platform than the one used for training the model. The system and hardware requirements for these stages are very different.
+
+The Unity game template we prepared uses the Unity Machine Learning (ML) Agents toolkit. This toolkit provides tools for both training and inference. It also supports two types of machine learning: _Imitation Learning_ and _Reinforcement Learning_.
+
+Imitation Learning uses an "expert" for the AI to learn from while Reinforcement Learning uses rewards and punishments.
+
+We will focus on Reinforcement Learning.
+
+Let's start by installing the necessary tools before diving deeper.

content/learning-paths/smartphones-and-mobile/using_unity_machine_learning_agents/03-Install.md

@@ -0,0 +1,83 @@
+---
+title: Install Unity and the project
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Getting started
+
+The Unity ML Agents toolkit provides a C# API for use in your project and Python scripts to run outside of Unity. We'll need Python itself and a few extra libraries.
+
+We need the Python tools before we can run our training stage, but to get going quickly we'll just install Unity for now.
+
+## The tools we will need
+
+We will install Unity via the Unity Hub. The Unity Hub provides an easy way to manage multiple installations of Unity. It also provides a straightforward way of adding the extra support modules we'll need. Here is what we'll end up with:
+
+* Unity Hub
+* Unity
+* Visual Studio Community Edition (Unity support module)
+* Dr Arm Unity project files (the accompanying zip file)
+* [_only for deploying to an Android device_] Android build support (Unity support module)
+
+The installation instructions in the next section will take us through the above steps one by one, starting with the Unity Hub.
+
+## Installing Unity Hub, Unity and support modules
+
+Assuming you don't already have Unity installed:
+
+1. Download and install the [Unity Hub](https://unity3d.com/get-unity/download?ref=personal)
+
+    Note that Personal licenses are free, and if running on Windows you will need Windows 10 or 11 and a GPU with DirectX 10 capabilities. If running Unity on a Mac or Linux, please follow the instructions for your platform.
+
+1. In your Unity Hub download and install Unity. The latest LTS (Long Term Support) release should work but this project has been tested on 2022.3.12f1.
+
+    1. In Unity Hub go to **Installs** and click **Install Editor**
+
+    1. Go to the _Archive_ page and find version 2022.3.12f1. Select the "Unity Hub" download button rather than downloading the Unity Editor directly
+
+    1. The install will require several gigabytes (including Android deployment options) so please check your available disk space first
+
+1. Add Android Build support
+
+    1. Click the _Settings_ icon and choose _Add Modules_
+
+        ![Unity Hub Installs](images/UnityHubInstalls.png "Figure 1. List of Unity versions installed")
+
+    1. Under **Dev Tools**, choose _Microsoft Visual Studio Community 2022_ if you wish to install it. This integrates well with Unity as a code editor, but any editor will do for editing scripts
+
+    1. Under **Platforms**, choose _Android Build Support_. Also make sure to choose _OpenJDK_ and _Android SDK & NDK Tools_ as well
+
+        ![Unity Hub Add Modules](images/unity-hub-add-modules.png "Figure 2. Tick the modules you want installed")
+
+    1. Press _Continue_, and then agree to the Android SDK and NDK License Terms to install everything
+
+    1. If you prefer to use an alternative script editor, you might want to check your editor is active; see section below _Check active script editor in Unity_
+
+## Download and open the Dr Arm Unity project
+
+1. Download and extract the accompanying Unity project from the [supporting zip file](../files/MLAgentsWorkshopSupportingFiles.zip).
+
+1. Open Unity Hub
+
+1. From the Unity Hub, under the Projects tab, click the down-arrow next to Open and select "Add project from disk".
+
+1. Choose the location of your workshop files (the directory that contains the _Assets_ folder)
+
+1. The project will appear in the list. Click it to open the project in Unity.
+
+1. It may take a few minutes for Unity to process the asset files.
+
+## Check active script editor in Unity
+
+Depending on your setup, it is possible that the default script editor is not set to your preferred editor:
+
+1. In Unity select _Edit_ menu and then _Preferences_
+
+1. In the _Preferences_ window, select _External Tools_ on the left
+
+1. Select your desired editor from the drop-down menu next to _External Script Editor_
+
+    ![External script editor window](images/unity-external-script-editor.jpg "Figure 3. External tool options in Unity")

content/learning-paths/smartphones-and-mobile/using_unity_machine_learning_agents/04-The-Dr-Arm-Game.md

@@ -0,0 +1,52 @@
+---
+title: The Dr Arm game
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## The Dr Arm Game
+
+This learning path is based on the video workshop [Using Unity's Machine Learning Agents on Arm](https://on-demand.arm.com/flow/arm/devhub/sessionCatalog/page/pubSessCatalog/session/1681291098539001B22U). We will go through all the key points from the video and use the same Dr Arm game project files (provided in the supporting zip file you extracted earlier).
+
+In the game, you control a fighting character (a Paladin) in battle against an AI opponent (a Vampire).
+
+The battle takes place in a small 3D arena. Both characters can perform the following actions:
+
+* Move (left, right, forward, backward)
+* Roll
+* Sword attack
+* Fireball attack
+
+Use the on-screen touch controls to control your character on mobile. The AI opponent is driven by an AI "brain" created with the ML Agents Toolkit.
+
+## Open and test the game
+
+Before diving into machine learning in Unity, let's first look at the game. We have a ready-to-play version in the Unity scene called _Level_DevSummit2022_ReadyToPlay_.
+
+How a model was trained and how this pre-trained model was plugged into and used within the game are all explained in detail in later sections of this learning path.
+
+Dr Arm is intended for mobile play but we can test it within the Unity editor first to ensure everything is present and correct. To do this, we'll run the game in demo mode, which shows two AI characters battling each other.
+
+Assuming you are using the default editor layout:
+
+1. Navigate to the _Assets/#DevSummit2022/Scenes_ directory in the Project panel
+
+1. Double-click the scene file _Level\_DevSummit2022\_ReadyToPlay_ (depending on your platform or settings, it may show the _.unity_ file extension)
+
+1. The scene should open inside the _Scene_ tab
+
+1. Click the play button
+
+1. You may some some warnings in the _Console_ tab but these can be ignored as we haven't done all the machine learning steps yet
+
+1. You should be presented with a simple title screen with a menu of _Easy_, _Medium_, _Hard_ and _Demo_ options
+
+![Dr Arm Title Screen](/images/game-title-screen.jpg "Figure 1. Title screen with menu options")
+
+1. Click _Demo_ to run a battle between two AI characters
+
+![Dr Arm Demo Mode](/images/game-demo-mode.jpg "Figure 2. Demo mode running two AT characters against each other")
+
+The screenshots above show a working version. If you see something similar, you know everything is working as expected. In the next section, we'll dive into machine learning and how we leverage ML in Dr Arm.

content/learning-paths/smartphones-and-mobile/using_unity_machine_learning_agents/05-Machine-learning-in-Unity.md

@@ -0,0 +1,94 @@
+---
+title: Machine Learning in Unity
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Overview of machine learning
+
+One way to implement a machine learning system is using _Neural Networks_. This is the method enabled by Unity ML Agents.
+
+In this method, we train a neural network model (a "brain") that takes inputs (observations), processes them, and produces outputs (decisions). These outputs drive the actions of the AI characters in our game.
+
+Neural networks are a vast and complex subject. To simplify, they contain inputs (input nodes), outputs (output nodes), and internal nodes that process the input values. Input values get modified as they pass through the internal nodes to reach the output nodes.
+
+Internal nodes have "weights" which are numerical parameters that affect a node's output to subsequent nodes. We train a neural network model by feeding sample input values, inspecting the output values, and comparing the output to expected values over many iterations. We change these node weights until satisfied with the results.
+
+Neural networks with more than one layer of internal nodes are called _Multilayer Perceptrons_ (MLPs).
+
+In a game, the outputs of such a model directly control an AI character's actions, which affect the data for each training iteration. It can take thousands of iterations before results are satisfactory and usable in a real game.
+
+This computationally intensive process runs during the training stage only. We prepare the data (the model) before building and deploying the game. The training produces the model in the form of a data file that we embed in the game, loaded at runtime into our game's ML subsystem.
+
+The goal of training is a network that produces desirable outputs (sensible decisions leading to sensible actions) from sensible inputs (observations).
+
+As you can imagine, training the model becomes a separate new build step before bundling and deploying the game. Carefully choosing observations and actions is key to AI quality.
+
+![Neural network](images/mlp-network.png "Figure 1. Neural network diagram")
+
+## What is Unity ML Agents toolkit?
+
+The Unity ML Agents toolkit is an open source machine learning framework created and released by Unity Software Inc.
+
+Written in C# and Python, it contains APIs and tools for training AI "Agents". On the Unity side, we have _Agents_ we train that provide observations and actions. The _Agents_ connect to the Python framework (_mlagents-learn_ program) which provides algorithms for _Reinforcement Learning_.
+
+The training step produces a data file which would be our pre-trained model. This model is then embedded into our game and used at runtime to act as "the brain" of our AI character (the _Agent_).
+
+## Dr Arm and Reinforcement Learning
+
+Reinforcement learning is a type of machine learning where an "agent" is trained with the goal of maximizing some reward and/or minimizing some punishment.
+
+Rewards and punishments are scores (floating point numbers). Inputs are "observations" of the environment and opponent (also turned into floating point numbers).
+
+### Our reward function
+
+The reward tells the character if actions succeeded (positive) or failed (negative). This allows characters to learn to do the right thing - in our case, quick victories.
+
+We reward with a value of 1 for winning and -1 for losing. A small negative reward accumulates over time and reduces the reward for slower wins. This will incentivize quick wins during training iterations.
+
+### The inputs
+
+There are a number of inputs that we will use - each input is in the form of a floating point number. Our inputs include:
+
+* Health, stamina and mana level of both characters
+* Is the character rolling?
+* Is the enemy rolling?
+* Is the enemy facing the character?
+
+### The outputs
+
+The outputs from the neural network model directly translate to actions the character can perform. There are 3 outputs in total:
+
+* A floating point value between -1 and 1, used as joystick input X
+* A floating point value between -1 and 1, used as joystick input Y
+* An integer value that will decide whether the character should attack, roll or use fireball (or none of them).
+
+### Unity training scene
+
+We have a dedicated scene that runs 6 battles simultaneously. Both characters are AI. This reduces training time.
+
+## How does the training work?
+
+First off, projects need to implement all the necessary hooks for Unity ML Agents to function. We have scripts for our "intelligent" characters that derive from the _Agent_ class and call the necessary ML-related functions at the correct times:
+
+1. Setup training limits such as maximum simulation steps (so we don't allow a battle to go on forever)
+1. Send observations (inputs) to the agent
+1. Receive outputs from the agent (as described above)
+1. Character performs actions based on the output from the agent
+1. Send reward (positive or negative) to the agent
+
+Training with Unity ML Agents toolkit leverages the Unity project itself in the training. Here is a quick overview of the steps:
+
+1. Open your training scene in Unity
+1. Run the _mlagents-learn_ python process
+1. Launch your game in the Unity editor (if you wait too long the connection may fail)
+1. ML Agents Toolkit will automatically connect and run the simulation
+1. You can monitor the training by using TensorBoard (described later)
+
+Training neural networks is computationally intensive and can take a long time (hours to potentially days), but can leverage GPU computing to reduce the time it takes.
+
+It is worth mentioning that the time it takes to train a neural network is different to the runtime performance of using a pre-trained ML model. Runtime performance is crucial for end users who will be playing the game. We have found Unity ML Agents runs very well on Arm devices.
+
+In the next section we'll go through the main Dr Arm scene and some of the key objects and ML components.