AI Big World

Maths for AI & ML

Wed, 25 Nov 2020 00:00:00 +0000

Slides

Gradient-based Optimization: A short introduction to optimization in Deep Learning by Christian S. Perone

Free Online Books

Tue, 24 Nov 2020 00:00:00 +0000

English

Project Gutenberg mostly English books

React Resources

Sun, 22 Nov 2020 00:00:00 +0000

Learning React

React Workshop by Ben Ilegbodu, a Christ follower

Seq2Seq + Attention (Minsuk Heo)

Sun, 22 Nov 2020 00:00:00 +0000

[딥러닝 기계번역] 시퀀스 투 시퀀스 + 어텐션 모델 youtube

Problem

How to translate a sequence of words to a target language

For instance, ‘I love you’ => ‘난 널 사랑해’

Note that the order of tokens is different. The problem is now how can you build a model that will translate in the correct order of the target language all without explicit rules.

Sequence to sequence

To apply it in translation tasks, you can use RNN to build the context vector that will output target language sequence. The Encoder-Decoder architecture can be used where the encoder packs the source sentence information into the context vector from which the decoder extracts the target sentence.

=> Problem with a fixed size context vector

For a long sequence, the context vector window size is not enough

Attention mechanism

We can use encoder’s each state with current state to generate dynamic context vector.

Encode info into a sequence of vectors not in a single context vector
Chooses a subset of the vectors adaptively while decoding the translation

Teacher forcing

When the prediction is wrong, force the target word (ground truth) as the next input to the decoder

results in a faster and more stable model

Reference

Neural Machine Translation by Jointly Learning to Align and Translate

Stanford: NLP with Machine Learning (2)

Sat, 21 Nov 2020 00:00:00 +0000

Lecture 2: Word Vectors 2 and Word Senses

2019 Winter | video | slides | notes

Lecture Plan

Lecture 2: Introduction and Word Vectors

Main idea of word2vec

Iterate through each word of the whole corpus
Predict surrounding words using word vectors
The probability distribution is found by the dot product of the word vectors by the softmax function

\[{U (outside)} {V (center)} {U.{v_{4}}^T} (dot product) {softmax(U.{v_{4}}^T) (probabilities)}\] \[\left[ \begin{array} ..... \\ ..... \\ ..... \\ ..... \\ ..... \\ ..... \\ \end{array} \right]_{U (outside)} \left[ \begin{array} ..... \\ ..... \\ ..... \\ ..... \\ ..... \\ ..... \\ \end{array} \right]_{V (center)} \left[ \begin{array} . \\ . \\ . \\ . \\ . \\ . \\ \end{array} \right]_{U.{v_{4}}^T} \left[ \begin{array} . \\ . \\ . \\ . \\ . \\ . \\ \end{array} \right]_{softmax(U.{v_{4}}^T) probabilities}\]

Same predictions at each position.
We want a model that gives a reasonably high probability estimate to all words that occur in the context.
Word2vec maximizes objective function by putting similar words nearby in space.

2. Optimization: Gradient Descent

We have a cost function $J(\theta)$ to minimize
Gradient Descent is an algorithm to minimize $J(\theta)$
Idea: for current value of $\theta$, calculate gradient of $J(\theta)$, then take a small step in the direction of negative gradient & repeat.

Gradient Descent - updated equation

(in matrix rotation)

$\theta^{new} = \theta^{old} - \alpha{\nabla}_\theta J(\theta)$

$\alpha$ = step size or learning rate
(for a single parameter)
\[\theta_j^{new} = \theta_j^{old} - \alpha\frac{\alpha}{\alpha\theta_j^{old}} J(\theta)\]

Algorithm

while True:
  theta_grad = evaluate_gradient(J, corpus, theta)
  theta = theta - alpha * theta_grad

Problem: $J(\theta)$ is a function of all windows in the corpus (potentially billions!) So $\nabla_{\theta} J(\theta)$ is very expensive to compute.
Solution: Stochastic gradient descent (SGD)
- Repeatedly sample windows and update after each one

Algorithm:

while True:
  window = sample_window(corpus)
  theta_grad = evaludate_gradient(J, window, theta)
  theta = theta - alpha * theta_grad

Iteratively take gradients at each such window for SGD
But in each window, we only have at most 2m + 1 words so $\nabla_{\theta} J_t(\theta)$ is very sparse!
We might only update the word vector that actually appear!
Solution: either you need sparse matrix update operations to only update certain rows of full embedding matrices U and V, or you need to keep around a hash for word vectors
If you have millions of word vectors and do distributed computing, it is important to not have to send gigantic updates around!
Why two vectors? –> Easier optimization. Average both at the end
- But can do algorithm with just one vector per word
Two models
1. Skip-grams: Predict context (outside) words (position independent) given a center word
2. Continuous Bag of Words (CBOW): Predict center word from (bag of) context words
For additional efficiency: negative sampling Train binary logistic regressions for a true pair (center word and word in its context window) versus several noise pairs (the center word paired with a random word)

Reference

Stanford NLP with Deep Learning by Chris Manning
- videos
- New online certificate course in 2021
- Chris Manning’s github Text Analysis for Humanities Research
Distributed Representations of Words and Phrases and their Compositionality (Mikolov, et al. 2013) NeuIPS

Experimental Creative Writing with the Vectorized Word

Sat, 21 Nov 2020 00:00:00 +0000

Experimental Creative Writing with the Vectorized Word

Sept 28-30, 2017 at thestrangeloop.com

youtube
Understanding word vectors github

Allison Parish is a poet and programmer and currently a member of the full-time faculty at NYU’s Interactive Telecommunications Program.

Poetry is just a language that calls for its own aesthetic properties.
The distributional hypothesis: “linguistic items with similar distributions have similar meanings.”
GloVe (“Global vectors for word representations”)
- github
CMU Pronouncing Dictionary

NLP Events

Sat, 21 Nov 2020 00:00:00 +0000

EMNLP 2020

Empirical Methods in NLP 2020

Simultaneous Translation by Liang Huang, Colin Cherry, Mingbo Ma, Naveen Arivazhagan, Zhongjun He
- prefix-to-prefix, not seq-to-seq

Business As Mission

Sat, 21 Nov 2020 00:00:00 +0000

Training programs

Repurposing Business by Brett Johnson
- facebook

Neurodivere News and Information

Sat, 21 Nov 2020 00:00:00 +0000

Reference

Dandelion Program - Cornell University
- DXC Dandelion Program Australian program for neurodiversity.

Markdown Tips and Tricks

Fri, 20 Nov 2020 00:00:00 +0000

How to create the table of contents

This cannot be done automatically unfortunately, but you can do manually by adding hyperlinks for each section and use the section ids to create a table of contents.

# Table of Contents
1. [Introduction](#introduction)

Then in the corresponding section, add an embedded hyperlink:

# Introduction <a name="introduction"></a>

subscript and superscript

Add the tags sub or sup tags x_t²

x<sub>t</sub><sup>2</sup>

bold, italics, and bold italics

Either use surrounding underscores or asterisks

italics bold bold italics

_italics_
__bold__
___bold italics___

*italics*
**bold**
***bold italics***

Math formulae

There are different ways of displaying math formulae.

NB: Use an online math editor like an online LaTex editor to check markdown and latex syntax.
- List of LaTex math symbols
- LaTex editor

1. Use mathjax header (preferred)

Add a script header at the top of the markdown file

<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>

Then write your formula like these:

$$
M = \left( \begin{array}{ccc}
x_{11} & x_{12} & \ldots \\
x_{21} & x_{22} & \ldots \\
\vdots & \vdots & \ldots \\
\end{array} \right)
$$

$$x_{1,2} = \frac{-b \pm \sqrt{b^2-4ac}}{2b}$$

They are rendered separately as:

\[M = \left( \begin{array}{ccc} x_{11} & x_{12} & \ldots \\ x_{21} & x_{22} & \ldots \\ \vdots & \vdots & \ldots \\ \end{array} \right)\] \[x_{1,2} = \frac{-b \pm \sqrt{b^2-4ac}}{2b}\]

2. Latex image tags

Inserting an img html tag with the formula

<img src="https://render.githubusercontent.com/render/math?math=e^{i \pi} = -1">

will be rendered as:

A similar method is:

![\Large x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}](https://latex.codecogs.com/svg.latex?x%3D%5Cfrac%7B-b%5Cpm%5Csqrt%7Bb%5E2-4ac%7D%7D%7B2a%7D)

Jekyll Tips and Tricks

Fri, 20 Nov 2020 00:00:00 +0000

Files and folders

File names

A blog file needs to have the format of [year]-[month]-[day]-[name of the post].md

Folder location

The blog files need to be under _posts folder, but you can create sub-folders and they will be picked up by Jekyll as a sub-path in url. For instance, if you have ‘ai’ as s sub-folder, the url path will be

https://https://changsin.github.io/ai/2020/11/20/<blog name/

Image files

If you want to have images in your blog post, you need to have the images in a sub-folder. The problem is the location of the image files need to be correctly specified. For Jekyll, each blog post is mapped differently depending on the date. The best way is to create a static sub-folder under assets and create links.

    ![word2vec](https://changsin.github.io/assets/images/wordvec_overview.png)

Stanford: NLP with Machine Learning (1)

Fri, 20 Nov 2020 00:00:00 +0000

Lecture 1: Introduction and Word Vectors

2019 Winter

video | slides | notes

Lecture Plan

Lecture 1: Introduction and Word Vectors

The course (10 mins)
Human language and word meaning (15 mins)
Word2vec introduction (15 mins)
Word2vec objective function gradients (25 mins)
Optimization basics (5 mins)
Looking at word vectors (10 mins or less)

How do we represent the meaning of a word?

1. As discrete symbols

“denotational semantics”: commonest linguistic way of thinking of meaning

  signifier (symbol) ⟺ signified (idea or thing)

aka. representational theory of meaning

How do we have usable meaning in a computer? Common solution: WordNet, a thesaurus containing Lists of synonym lists and hypernyms (“is a” relationships)
- Problems
  - Great as a resource but missing nuance
  - Missing new meanings of words
  - subjective
  - Requires human labor to create and adapt
  - Can’t compute accurate word similarity

Representing words as discrete symbols

In traditional NLP, we regard words as discrete symbols: a localist representation: “hotel”, “conference”, “motel”, etc.

Words can be represented by one-hot vectors

motel = [ 0 0 1 0 ]
hotel = [ 0 1 0 0 ]

Problems The two vectors are orthogonal, no natural notion of similarity.

Solution:

Could try to reply on WordNet’s list of synonyms to get similarity?
- But it is well-known to fail badly: incompleteness, etc.
Instead: learn to encode similarity in the vectors themselves

Representing words by their context

** Distributional semantics***: A word’s meaning is given by the words that frequently appear close-by

One of the most successful ideas of modern statistical NLP!

Word vectors

Build a dense vector for each word, chosen so that it is similar to vectors of words that appear in similar context.
word vectors are sometimes called word embeddings or word representations. They are distributed representations.
Word2vec (Mikolov et al. 2013) is a framework for learning word vectors
Idea
- We have a large corpus of text
- Every word in a fixed vocabulary is represented by a vector
- Go through each position t in the text, which has a center word c and context (“outside”) words o
- Use the similarity of the word vectors for c and o to calculate the probability of o given c (or vice versa)
- Keep adjusting the word vectors to maximize this probability.
Objective function For each position $t = 1, ..., T$, predict context words within a window of fixed size m, given a center word $w_{j}$

\[L (\theta) = \prod_{t = 1}^{T} \prod_{ -m \leq j \leq m \hspace{0.8mm} (j \neq 0) } P(w_{t+j}|w_{t};\theta)\]

$\theta$ is all variables to be optimized

The objective function $J(\theta)$ is the (average) negative log likelihood:

\[J (\theta) = -\frac{1}{T}logL(\theta) = \prod_{t = 1}^{T} \prod_{ -m \leq j \leq m \hspace{0.8mm} (j \neq 0) } log P(w_{t+j}|w_{t};\theta)\]

Question: How to calculate $P(w_{t+j} \| w_{t};\theta)$?
- Answer: We will use two vectors per word w:
  - $v_{w}$ when w is a center word
  - $u_{w}$ when w is a context word

Then for a center word c and a context word o:

\[P(o|c) = \frac{exp(u_{o}^{T}v_{c})}{\sum_{w \in V} exp(u_{w}^{T}v_{c})}\]

$u_{o}^{T}v_{c}$: dot product compares similarity of o and c. $u^{T}v = u v = \sum_{i=1}^n u_i v_i$ Larger dot product = larger probability
Exponentiation makes anything positive $exp(u_{o}^{T}v_{c})$
Normalize over entire vocabulary to give probability distribution $\sum_{w \in V} exp(u_{w}^{T}v_{c})$ This is an example of the softmax function $\mathbb{R}^n \rightarrow (0, 1)^n$

\[softmax(x_i) = \frac{exp(x_i)}{\sum_{j=1}^n exp(x_j)} = p_i\]

The softmax function maps arbitrary values $x_i$ to a probability distribution $p_i$
- “max” because it amplifies probability of the largest $x_i$
- “soft” because it still assigns some probability to smaller $x_i$
Train a model by optimizing parameters

To train a model, we adjust parameters to minimize a loss.

$\theta$ represents all model parameters in one long vector
In our case with d-dimensional vectors and V-many words

\[\theta = \left[ \begin{array}{c} v_{aardvark} \\ v_a \\ \vdots \\ v_{zebra} \\ u_{aardvark} \\ u_a \\ \dots \\ u_{zebra} \end{array} \right] \in \mathbb{R}^{2dV}\]

Remember: every word has two vectors
We optimize these parameters by walking down the gradient

Two model variants

Skip-grams (SG): Predict context (“outside”) words (position independent) given a center word
Continuous Bag of Words (CBOW): Predict center word from (bag of) context words

The lecture assumed Skip gram model so far.

Gensim word vector visualization
Exploring Word Vectors code

Reference

Stanford NLP with Deep Learning by Chris Manning
- videos
- New online certificate course in 2021
- Chris Manning’s github Text Analysis for Humanities Research
Distributed Representations of Words and Phrases and their Compositionality (Mikolov, et al. 2013) NeuIPS

Interesting ML Projects

Thu, 19 Nov 2020 00:00:00 +0000

130 ML Projects Solved and Explained

Coding Helper

AI DocString VS Extension that generates python docstring automatically

Latin quotes

Thu, 19 Nov 2020 00:00:00 +0000

fortuna favet audaci
(fortune favors the bold)

What is Word Embedding?

Wed, 18 Nov 2020 00:00:00 +0000

What is word embedding?

Word embedding refers to the process of converting words to numbers so that they can be used for machine learning.

Word encoding

A simple idea is to simply represent the whole vocabulary indexing each word as a position in an array. The result is a vector representation of each word.

word	encoding
king	[1, 0, 0, 0]
man	[0, 1, 0, 0]
queen	[0, 0, 1, 0]
woman	[0, 0, 0, 0]

A problem with this simple approach is that there is no similarity among related words. For instance, king and queen are similar in terms of kingship but they are simply separate points in the Euclidean space. We need a representation that somehow preserves similarity measures among words.

Word embedding

Word embedding is the hidden layer’s representation of a word and its context after training.

Embedding over encoding

Embedding is dense vector with similarity
Similarity comes from neighboring words

word	encoding	embedding
king	[1, 0, 0, 0]	[1, 2]
man	[0, 1, 0, 0]	[1, 3]
queen	[0, 0, 1, 0]	[5, 1]
woman	[0, 0, 0, 0]	[5, 3]

Word2Vec data generation (skip gram)
- windows size = 1

How to represent words?

Discrete representation: The vast majority of rule-based and statistical NLP work regards words as atomic symbols: hotel, conference, etc.
- WordNet? - subjective, requires human labor to create and adapt
- Simple idea: “One-hot encoded vectors” - very sparse vectors
- Build a vector for a large vocabulary of words
- Turn on the corresponding element for the word in the vector
- The problem is that there is no similarity among related words: e.g., Seattle motel vs. Seattle hotel
- localist representation
Distributional similarity based representations
- You can get a lot of value by representing a word by means of its neighbors.
- One of the most successful ideas of modern statistical NLP.
```
You shall know a word by the company it keeps.
  --J. R. Firth (1957)
```
```
Words that occur in similar context tend to have similar meanings.
    -- Harris (1954)
```
  word embeddings = distributional semantic model, distrubuted representation = semantic vector space = vector space model
  
  Similar to later Wittgenstein
Applications
- compute similarities between words
- create groups of related words
- use as features in text classification
- document clustering
- NLP tasks:
  - Part-of-Speech tagging
  - Sentiment Analysis
  - Syntactic parsing
Steps
1. Summarize the occurrence statistics for each word in a large document set
2. Apply some transformation to the counts: e.g., dimensionality reduction (SVD) to obtain dense real-valued vectors
3. Compute similarity between words as vector similarity: cosine similarity between words = angle between vectors
Predict methods
1. In one setup, the goal is to predict a word given its context.

Limitations

Compositionality: multi-word or sentee
Words with multiple senses
Tailed to a specific similarity measure: only synonyms
Topical similarity: usually captured using topic models

Word meaning is defined in terms of vectors:
- We will build a dense vector for each word type, chosen so that it is good at predicting other words appearing in its context… those other words also being represented by vectors…it all gets a bit recursive

distributed representation != distributional similarity

Basic idea of learning neural network word embeddings

We define a model that aims to predict between a center word w_t and context words in terms of word vectors.

p(context|w_t)

which has a loss function, e.g.,

J = 1 - p(w_-t|w_t)

-t means all other context words

We look at many positions t in a big language corpus.

We keep adjusting the vector representations of words to minimize this loss.

Directly learning low-dimensional word vectors

Learning representations by back-propagating errors (Rumelhart et al. 1986)
A neural probabilistic language model (Bengio et al., 2003)
NLP (almost) from Scratch (Collobert & Weston, 2008)
A recent, even simpler and faster model: word2vec (Mikolov et al. 2013) –> intro now

Main idea of word2vec

predict between every word and its context

Two algorithms

Skip grams: Predict context words given target (position independent)
Continuous Bag of Words (CBOW): Predict target word from bag-of-words context

Two (moderately efficient) training methods

Hierarchical softmax
Negative sampling

For each word t = 1 … T, predict surrounding words in a window of “radius” m of every word.

Objective function: Maximize the probability of context word given the current center word:

\[J'(\theta) = \prod_{T}^{t=1}\]

Minsuk Heo’s word2vec tensorflow tutorial
Word Similarity
Minsuk Heo 자연어처리의 유사도 측정 방법(거리측정, 코사인 유사도)
Euclidean distance - using Pythagorean theorem
Cosine similarity - based on the angles - ignoring vector magnitude

$similarity = cos(\theta) = \frac{A \cdot B}{||A|| \; ||B||}$

$$ A $$ is the distance of A
The angle between two terms cannot be greater than $90^{\circ}$ because a word count being a negative number doesn’t make any sense. (if you plot word frequencies between two words as x & y coordinates)

TF Hub Word2Vec

Minsuk Heo TF Hub Word2Vec
- github

Train a model to predict neighbor words from the current word and vice versa.
Use trained model’s hidden layer as word embedding.

Word2Vec by Jordan Boyd-Graber

Jordan Boyd-Graber Understanding Word2Vec
similarity is calculated using cosine similarity: $sim(d\overrightarrow{o}g, c\overrightarrow{a}t) = \frac{d\overrightarrow{o}g \cdot c\overrightarrow{a}t} {||d\overrightarrow{o}g|| \; ||c\overrightarrow{a}t||}$
Finding the most similar words to $d\overrightarrow{o}g$
- Compute the similarity from word $\overrightarrow{v}$ to all other words
- This is a single matrix-vector product: $W \cdot \overrightarrow^{T}$
- Result is a |V| sized vector of similarities
- Take the indices of the k-highest values
- FAST! for 180k words, d=300: ~30sm
Similarity to a group of words
- Find me words most similar to cat, dog and cow”.
- Calculate the pairwise similarities and rum them: $W \cdot c\overrightarrow{a}t + W \cdot c\overrightarrow{a}t$
- Now find the indices of the highest values as before
- Matrix-vector products are wasteful. Better option: $W \cdot (c\overrightarrow{a}t + d\overrightarrow{o}g + c\overrightarrow{o}w)$

Reference

Efficient Estimation of Word Representations in Vector Space (Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean)
Stanford NLP with Deep Learning
Udacity TensorFlow word2vec
tmikolov

RNN

Sun, 15 Nov 2020 00:00:00 +0000

Text Generation

English

TensorFlow tutorial Text Generation
TensorFlow tutorial Neural machine translation with attention Spanish and English translation
David Foster’s LSTM Text Train using Aesop’s THE FOX AND THE GRAPES
Edureka RNN tutorial by Anirudh Rao
NYU Deep Learning sequence classification
MIT Deep Learning Music Generation
Stanford Music Composition using RNN pdf
Karpathy’s Char RNN - in Lua using Tiny Shakespeare
Sachi Parikh’s TV Script Generator using Simpson’s episode script

Chinese

Generating Chinese Classical Poems with RNN Encoder-Decoder

Korean

RNN을 이용한 한국어 감성분석 : 온라인 영화 후기를 중심으로

Translation

How to build a translation pipeline with RNN and Keras Spanish-English translation.
- Jupyter notebook

AI & Deep Learning Concepts

Sat, 14 Nov 2020 00:00:00 +0000

PyTorch Loss Functions

Interesting GAN Projects

Sat, 14 Nov 2020 00:00:00 +0000

GAN

SkyAR

a vision-based method for video sky replacement and harmonization, which can automatically generate realistic and dramatic sky backgrounds in videos with controllable styles.

paper: Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos
code

Faceapp

換臉照怎麼做的？當紅Faceapp背後AI技術大解析

Magic Sand

Magic Sand by Thomas Wolf: a software for operating an augmented reality sandbox.

Face Generation

GAN Face Generation high school student project

Python Tips and Tricks

Thu, 12 Nov 2020 00:00:00 +0000

1. General

How to check the version of python module

pip freeze

2. JSON tips and tricks

To serialize into json

import json

python_object = dict()

json_dump = json.dumps(python_object)

To json file

def to_file(file_name, data):
    with open(file_name,  'w', encoding="utf-8") as json_file:
        json_file.write(data)

To deserialize from json

import json

json_string = "{}"

json_object = json.loads(json_string)

From a json file

import json

def from_file(path):
    try:
        f = open(path, "r", encoding="utf-8")
        data = json.load(f)
        f.close()
        return data

    except Exception as ex:
        print(ex)

How to setup CVAT

Mon, 09 Nov 2020 00:00:00 +0000

docker ps after running docker-compose up -d

CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                  NAMES
c59396ebc695        nginx:stable-alpine   "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes        0.0.0.0:8080->80/tcp   cvat_proxy
9adce835f4db        cvat/ui               "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes        80/tcp                 cvat_ui
c09a03e4cd06        cvat/server           "/usr/bin/supervisord"   2 minutes ago       Up 2 minutes        8080/tcp, 8443/tcp     cvat
17b863adf3bf        postgres:10-alpine    "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        5432/tcp               cvat_db
ac12ddb3ea81        redis:4.0-alpine      "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        6379/tcp               cvat_redis

Enter into the db container
```
docker exec -it cvat_db bash
```
Create user testworks
```
su - root -c "createuser testuser"
```
Connect to the db
```
docker exec -it cvat_db psql cvat
```

Grant all privileges

grant all privileges on database cvat to testuser;

NLP Projects

Sun, 08 Nov 2020 00:00:00 +0000

The Super Duper NLP Repo

Text Generation with an RNN TensorFlow tutorial

RNN

BERT

commit-autosuggestions: interesting project done by an undergraduate to auto-suggest commit messages based on CodeBERT: A Pre-Trained Model for Programming and Natural Languages
CodeBERT

GPT-2

GPT-3

GPT-3: A Hitchhiker’s Guide

Data

Webster’s English Dictionary

Cracking the Creativity Code

Sun, 08 Nov 2020 00:00:00 +0000

Cracking the Creativity Code: Discovering Ideas

This is the first part of two courses. The first part is “discovering” ideas and the second part delivering the ideas.

Coursera: Cracking the Creativity Code: Discovering Ideas
Matryoshka dolls - small to big
Cracking the Creativity Code: Zoom Out/Zoom In/Zoom Out Framework for More Creativity.
Creativity is widening the range of choice.
Ideas are “discovered”, not “created”.

For most of us, the problem isn't that we aim too high and fail.
 It's just the opposite. We aim too low and we succeed.
  -- Ken Robinson

Human individuals live far within their limits.
  -- William James

Motivation is the key.

The surprising fact is that all the variables which discriminate
 between creative and non-creative people are motivational.
-- John Hayes (1990)

Da Vinci’s 8 Power Questions

What is your deepest passion?

When am I most ‘myself’?
What is the ONE THING I could do, or stop doing, that would most improve the quality of my life?
What is my greatest talent?
How can I get paid for doing what I love best?
Who are my most inspiring role models?
How can I change the world?
What is my deepest passion?
What will be my life’s legacy?

Simon Sinek: Start with Why!

Start with why – how great leaders inspire action

The golden circle: what, how, and why Communicate inside out, not outside

Apple example

How other companies communicate

We make great computers.
They are beautifully designed, simple to use and user friendly.
Wanna buy one?

How Apple communicates

Everything we do, we believe in challenging the status quo. We believe in thinking differently.
The way we challenge the status quo is by making our products beautifully designed, simple to use and user friendly.
We just happen to make great computers.

People don’t buy what you do. They buy why you do it.

What you do simply proves what you believe.
People will do the things that prove what they believe.
When you communicate from inside out, we’re talking directly to the part of the brain that controls behavior, and then we allow people to rationalize it with the tangible things we say and do.
The goal is not to sell to people who need what you have. It is to sell to people who believe what you believe. The goal is not just to hire people who need a job. It is to hire people who believe what you believe.

Wright brothers vs. Samuel Pierpont Langley

Wright brothers were driven by a cause to change the course of history
Langley was in pursuit of the result, in pursuit of the riches.
The law of diffusion of innovation
- Needs between 15 and 18 percent market penetration.
- 10% -> 15% “crossing the chasm” (Jeffrey Moore)
TiVo failure
- What you do simply serves as the proof of what you believe.

Martin Luther King Jr. “I have a dream” speech

I believe - They showed up for what they believed.
not “I have a plan” speech
We follow the inspiring leader not because we have to, but because we want to. We follow those who lead, not for them, but for ourselves.

How to Start a Startup

Sun, 08 Nov 2020 00:00:00 +0000

How to start a startup

Sam Altman, President of Y Combinator, and Dustin Moskovitz, Cofounder of Facebook, Asana, and Good Ventures, kick off the How to Start a Startup Course. Sam covers the first 2 of the 4 Key Areas: Ideas, Products, Teams and Execution; and Dustin discusses Why to Start a Startup.

How to Start a Startup: youtube channel. Stanford lectures.
lecture trascripts
Slides
- Lecture 1 Sam
- Lecture 1 Dustin

Lecture 1 - How to Start a Startup (Sam Altman, Dustin Moskovitz)

Lecture video

1. Idea

Spend some time to think through ideas, the long term effect.
The idea comes first; the startup comes second.
Best companies are mission-oriented.
- The company should feel like an important mission.
- Good startups take ten years.
- Derivative ideas do not excite people.
Best ideas seem terrible at the beginning.
- Need conviction in your own ideas.
- Don’t need to worry about sharing your ideas.
- Unpopular but right ideas.
The growth of the market is more important than the growth of the company.
- Can’t create a market
Why now? – Sequoia
- Build something you need
If it takes more than a sentence to explain, it’s not a good idea.
Think about the market first.

2. Product

Build something users love.
Talk to users.
Better to build something small users love than something a lot of people like.
- The amount of love is the same. It’s just about how it is distributed.
Don’t worry about competitors.
It takes some fanatics to build a great product.
Pick a small number of users by hand.

Lecture 2 - Team and Execution (Sam Altman)

Lecture 2 video

3. Team

A. Cofounders: relentlessly resourceful. “James Bond”

Know your cofounders - tough & calm

B. Try not to hire: have a high bar. Get a small number of dedicated people.

C: Get the best people: 0 - 25% of your time.

Mediocre engineers do not build great companies.
Hire from people you already know (referrals).
Go for the aptitude, not experience in early days.
- Are they smart?
- Do they get things done?
- Do I want to spend a lot of time around?
- Good communication skills
- Manically determined
- Pass the animal test (Paul Graham): be able to describe as an animal
- Would feel comfortable reporting to them. Enjoy working with them. Or at least respect them.
Equity: be ready to give up to 10% to employees

D. You’ve hired the best - now keep them around.

What motivates people: autonomy & mastery

E. Fire fast

persistently negative

4. Execution

CEO jobs
- Set the vision
- Raise money
- Evangelize
- Hire and manage
- Make sure the entire company executes
Can you get it done?
- Focus:
  - What are you spending time and money on?
  - What are the two and three most important things?
  - Say no. A lot.
  - Set overarching goals. Repeat them.
  - Communicate.
- Intensity:
  - Relentless operating rhythm
  - Obsession with Execution Quality
  - Bias towards action - Indecisiveness is a startup killer.
    - Every time you talk to them, they’ve gotten new things done.
    - Do it incrementally
    - Quick
    - Do whatever it takes
    - Show up
    - Don’t give up
Always keep momentum: keep growing
- Save vision speeches for when you are winning. When you are losing, you just have to have some small wins. Sales fix everything.
- Shipping product
- Launching new features
- Reviewing/reporting metrics regularly
```
 The competitor to be feared is one who never bothers about you at all,
but goes on making his own business better all the time.
-- Henry Ford 
```

JavaScript Dev Tools

Sat, 07 Nov 2020 00:00:00 +0000

To develop interactively online

codepen: html, css, and js all in one editor online

Next.js free training

Mastering Next.js

What is Kaggle?

Sat, 07 Nov 2020 00:00:00 +0000

Kaggle is an online community for data scientists and machine learning practitioners. It provides public datasets, tools, and resources to learn and collaborate with others.

Korean hate speech detection

Korean hate speech detection

MNIST

CNN Keras

How Autoencoders work?

How autoencoders work?

KIC TRT Bootcamp Day3

Fri, 06 Nov 2020 00:00:00 +0000

When, Why, and How to Pivot
Product and Company Road Map
Funding

When, why and how to pivot

The tyranny of urgency

Quarterly cadence - serves as an accountability mechanism
What is a Pivot?
- When a startup shifts business strategy to accommodate industry, market, or customer
- The process of direct and indirect feedback changing your business model
- Iterations vs. Pivots
  - Iterations are course corrections
  - Pivots are painful - we were in the tire business, we’re now in apps
  - Not course correction
  - Slack - originally a gaming company
  - Odeo -> Twiiter
- When to do a pivot?
  - Path to revenue is not there
  - Customer review interviews
When & Why
- When - 90-180 days
  - Get out of your head and get in front of the customer
- 50-100 customer interviews
- Build on email list
How to pivot
- Are you staying in the same TAM?
- Is the product wrong?
- Is the brand wrong?
- Focus on the problem first, solution second!

Product and Company Road Map

Product vision
- Long term features
- Startup with PROBLEM not product - pivot product/features around the problem
- It won’t be what you ship first
- What the world looks like before and after your product
Product roadmap
- What features and when
- Quarterly targets based on
  - Budget
  - Available developers
Estimating costs
- Write specification
  - Required for launch
- Design & flow
  - Post project on fiverr.com
  - PPT/keynote first with design
- Test design and flow
- Revise
Company roadmap
- Roadmap is NOT dependent on fundraising
  - If you raise the capital, you move things forward in the roadmap
- Where will you be in 18 and 36 months
- Does it map to your financial model
- Maps to use of proceeds
Highlight key milestones
- This is a bit of an eye chart, so highlight key milestones
- Product ship dates
- First revenue
- Key hires: Hire two packs - for sales
- Quarterly Template

Funding 101

Fundraising fundamentals
- Raising funds is not a sign of success. Cash flow is.
- Control terms
- Convertible debt
- Needs a lead investor
Fundraising 101
- Stage appropriate capital = what’s the right money at the right time?
  - Early - angels, early stage VC’s
  - Later - large venture funds
- Control shifts - board roles, deal terms
- Dilution - every round of funding you will sell 35-35% of new shares
- Bad boards = bad enterprise value
Angels
- Invest in you and your passion
- Like the market or idea
- Have made money in the sector before
- Deal type
  - Convertible note with o cap
  - Revenue but pre-product makes fit
- “return the fund” deals that an produce > 10x
Later stage = growth capital
- Post product market fit
- $10k in marketing spend = internal rate of return (IRR) of how many days?
Fundraising process
- Enterprise sales process - landing multiple planes at the same time
- Top of the funnel
  - Research
    - Crunchbase (cheap) technation (UK)
    - Pitchbook (expensive)
    - Angelist
    - Competitive deals (w/ Exits)
  - Create a forwardable email
    - Ask for approval
    - Follow with intro
Target investors
- Targets - build of list of targets, look for linkedin and/or email introductions
- They have invested in your market/stage in the past
- They have board experience for your stage
- They can help with strategic introductions
  - Investors
  - Customers
Monthly updates
- Using MailChimp free account
  - Investors - tagged
  - Customers
- 30 day intervals
  - What we thought
  - What we learned
  - What we are doing about it
  - Where we need help
- Regularly
Do’s and Don’ts
- Do
  - Keep your eye on the ball
    - Don’t miss your numbers
  - Manage the process
  - Quality your investor
  - Create optionality
  - If there’s an elephant in the room, call it out
- Don’ts
  - Get < 6 months of capital
  - Hide the ball on any bad news
  - Be too transparent
  - Let the investor miss a deadline - escrow example
Fundraising process
- Asking for advice vs. cash
  - Who do you know that would like out type of deal
  - What other things would you like to see us complete before we talk again
- Deals that
Conclusion
- Broad funnels gives you the most choices
- They will be talking to other investors
- How you interact in the process will indicate the future
- What do you need in addition to capitol?
  - God to market help
  - Product help
  - Know your blind spots
- Time kills ALL deals
Tools for pitching
- Forwardable email
- Executive summary
- Presentation
  - ~13 slides
  - More is OK if you have data/traction Monthly update

KIC TRT Bootcamp Day2

Thu, 05 Nov 2020 00:00:00 +0000

How Startups Make Money
Pricing and Metrics That Matter
Go-To-Market
Traction and Product/Market Fit

How Startups Make Money

Do you know?

Cost to build
Cost to sell
Price
Profit

How NOT to make money?

Small transaction values
Small percentages
Not being able to capture value
Lumpy sales cycles
Long sales cycles
Pricing too low could kill your idea before you start

Marketing - Finding Customers

Hire someone who is culturally fit.

Messaging/Value proposition
- Does it resonate with the target customer?
- Is it repeatable?
- Don’t make your product the hero of the story - the customers are.
- Let them remember your story, not that you are smart.
Strategy
- Outbound first
- Inbound second

Key Metric Terms

CAC (Customer Acquisition Cost)
LTV (Customer lifetime value):
- How much is the customer worth over time.
- A prediction of the net profit attributed to the entire future relationship with a customer
- LTV to CAC ratio: 10 to 1 (B2B) or 5 to 1 (B2C)
ARPU (Average Revenue Per User): the total revenue divided by the number of subscribers. Used for subscription services.
Churn vs. Retention
Time to close
Customer engagement: how long do the users use?
Time on site

MQL & SQL

MQL (Marketing Qualified Lead)
- Define qualified vs list
- Suspects
- Prospects
- Qualified suspects
SQL (Sales Qualified Lead)
- Hand off to sales
- Identify steps to close
- Number of calls

Time to Close

Lead attribution
Source time
Compressing or expanding
Trend by source

Tracking

Data allows you to know where to double down and where to hold back
Conversion ratios

Sales - capturing value creation

How will they buy?
- Web Direct: place the order to buy on the web
- Direct: outbound sales, inside/outside
- Indirect/Channel: fulfills demand, doesn’t generate
- Retail

Business Development

Strategic relationships
Who has the list you want
How do you get the partnership
Strategy
- Who should you get as a partner: how will you build the relationship?
- What channels
Tactics
- Messaging/Value Proposition
Tools
- CRM

How to monetize your startup: Revenue Models

Business Models: abstract framework of creating, delivering, and capturing value
Revenue Models (part of business model) is the framework of revenue, pricing, who pays
Models Lean toward Tech and Product

1. Fee for Service

Example: Consulting Services company
Use: B2B & B2C
Key Metrics
- Project Revenue
- Cost of delivering (usually time)
- Gross Margin
Services are hard to scale because they require people to deliver including restaurant, contractors, etc.
Services include markup on cost of goods sold

2. Commerce

Example: Amazon, AmazonSupply
Use: B2C & B2B
Key Metrics:
- Wholesale or cost of goods sold
- Average Margin %
- Average Basket
Commerce: physical gooods - wholesale, cost of goods, retail, average margin, physical good
Notes: can mature into marketplace

3. Subscription

Example: salesforce, Box, Spotify
Use: B2C & B2B
Key Metrics
- Average Revenue Per User (ARPU)
- Conversion ratio - e.g., trial to purchase
- Churn
Challenges: MVP won’t be enough to be Kick Ass Product
Notes: Highest multiple, forecastable revenue

4. Metered service

Example: AWS, Splunk, Azure
Use: Favors B2B
Key Metrics
- Average Revenue Per User (ARPU)
- Conversion ratio - e.g., trial to purchase
- Churn
Challenges: MVP won’t be enough to be Kick Ass Product
Notes: Highest multiple, forecastable revenue

5. Transaction Fees/Rental

Example: 99Designs, Kickstarter, Elance, Chugg
Use: B2C & B2B
Key Metrics
- Average transaction revenue
- Fee % per transaction
- Number of transactions
Challenges: Margins are small (15%), need efficiency
Notes: Don’t start too low

6. Productize a Service

Your offerings are generally complex and requires services to deploy
- Gross margin on services >35%
- Product development comes with services
Use: B2C & B2B
Examples: Moz, service company convert to tools
Challenges: difficult to make the transition away from services

7. Combinations

Combinations business models happen for two reasons
- You don’t know which model is right
- At scale, you can expand revenue sources
Examples: Hardware sensors + software services to create data analytics
Challenges: most require scale or at least traction

8. Marketplaces

Example: eBay, Alibaba
Use: B2C & B2B
Key Metrics
- Average Transaction Amount
- Number of Monthly Transactions
- Commission %
Challenges: two sided market places require you to start with one side, value to seller & product market fit (x2)
Notes: critical mass or marketplace required

9. Lead Generation

10. Gaming

11. Advertising/Search

12. New Media

13. Big Data

14. Licensing

Pricing and Metrics That Matter

Pricing

Your price will be wrong! Increase it every year and grandfather early customers for a period of time

Art or science
- Benchmarks: Comparable products
- Don’t start too low
- Cost base vs. Value based
  - What is the alternate for the “Job to be Done”?
- Remember Freemium is a marketing mechanic, not a price
Product Pricing
- First Product
  - Staged pricing - up or down over time
- Second Product
  - Staged pricing - up or down over time
  - Launch timeline based on product road map
- Services?
  - Could it improve your sales cycle
Pricing Plan 30/60/90
- Rationalize your current price
- Can you move it up
- Can you bundle with other offers/services?
- Schedule quarterly pricing committee

Metrics that Matter

What to track - best practices
- Marketing metrics - “Marketing Motion”
  - Ad Spend
  - Traffic
  - Marketing Qualified Lead (MQLs)
- Sales metrics - “Sales Motion”
  - Outbound/Inbound
  - MQL to SQL conversion
- Business Development - partnerships What’s in it for them?
- Pricing

Go-to-market

Templates
- Four Templates
  - Marketplace
  - Subscription
  - Transaction Fee (works for commerce)
Financial Model Heuristics
- $0 - $40 is likely OK
- Start with assumptions
  - Product
  - Services
  - Release timing
- Scale revenue first
  - Customer count and timing
- Keep expenses in line as you grow

Traction and Product/Market Fit

Product Market Fit

Two epochs of every startup

Pre: the only thing that matters is getting there
- Some revenues
- Some customers
- some marketing
- A lot of hypothesis
Post: a whole new set of questions - scale, culture, and people
- Predictable & Scalable Revenue
- Customer profiles: Cohort analysis
- Churn
- Scalable marketing
- Hypothesis + Data
PMF Scorecard
- How would you feel if you could no longer use (product)? Disappoined?
Tracking Over Time
- Churn Rate
- Net Negative Churn
- NPS vs. CSAT
- Product usage level
  - Frequency
  - Recency
- Growth rate

Traction

Predictable/Forecastable Revenue
- De-risking your investment
- Predictable revenue moves you from valuations based on trailing 12 to future 12
- Target customer
- Sales Cycle
- Tools trail Strategy and Tactics
Customer Acquisition Hypothesis
- CAC
- LTV
- Time to close sale: how does this change with product/market maturity?
- Churn/Retention
- Average Revenue/Measure (user, account, etc.)
- Word of mouth vs. Virality

ML Conference EU 2020

Thu, 05 Nov 2020 00:00:00 +0000

Overview

Homepage: The Practical Machine Learning Conference

Day 1 (Nov. 5, 2020)

An Introduction to Transfer Learning in NLP and HuggingFace

Thomas Wolf

In this talk I’ll start by introducing the recent breakthroughs in NLP that resulted from the combination of Transfer Learning schemes and Transformer architectures. The second part of the talk will be dedicated to an introduction of the open-source tools released by HuggingFace, in particular our Transformers, Tokenizers and Datasets libraries and our models.

Computer Vision Using OpenCV

Beril Sirmacek

As an AI scientist and a developer, I have been engaged with AI-applications for many years especially focusing on object detection and recognition purposes. I love thinking that we can get creative in designing neural networks. We can train them supervised, unsupervised, semi or self-supervised, and this gives possibilities to mimic the human brain in a narrow domain. However, in vision applications, there are still things where AI is lacking and will be lacking without computer vision knowledge. Computer vision has been solving detection and recognition problems for many years. However, in the last decade, it seems like AI is seen as a replacement of computer vision. AI can find the optimal model for a specific type of data set and it might achieve generalization better. AI can be designed in a way that it can learn life-long which also brings possibilities of creating models which serve better when they are used longer. However, an AI vision system will be lacking capabilities without computer vision knowledge. First of all, it will require a very big data set to train the model what can be expensive or even not possible. On the other hand, computer vision systems can be modeled only by using a hand-drawn template image. Training AI models also requires GPUs. Nevertheless, I do not want to encourage everyone to train AI models for solving any simple problem which could be solved easily by computer vision. Last but not least, knowing computer vision, machine learning and especially feature engineering methods helps to design hybrid models that might be more robust to adversarial attacks or changing conditions.

In this lecture, I will briefly introduce how computer vision (especially using the OpenCV library) and machine learning can be used for creating detection and recognition models. Some experience with python, jupyter notebook and some machine learning background would be useful to get more benefits from this lecture.

The Evolution Revolution

Robert Plummer

Elegant and graceful mathematics make a cool textbook cover, but the inside of those same books are usually dry cold engineering. It’s important to mix the theory of innovation with the excitement of practicality, and through the composition of these elements we find innovation. In this talk, I’ll show you from an engineering perspective how to explore balance, and ultimately bottle machined success.

How to Machine Learn-ify any Product

Shivani Poddar Facebook

This talk will be a walkthrough of utilizing machine learning to replace a rule based system for consumers. We will discuss when is it okay to use ML, how to build these models with intelligent data, evaluate these offline and finally how to validate this evaluation to land these models in production systems. Furthermore, we will illustrate various self-learning/interactive-learning strategies that can be used for production systems to automate how models teach themselves to become better.

Teaching ML and AI to Coders

Laurence Moroney Google

Often it’s thought that to be able to succeed with Machine Learning and Deep Learning, as an onramp to Artificial Intelligence, that you need a deep background in mathematics and calculus, as well as some form of PhD. But you don’t. With modern APIs like TensorFlow, much of the complexity is abstracted away in pre-built libraries, so you can focus on learning. In this session, Laurence Moroney, from Google, will explain how he has used this to create courses with hundreds of thousands of students, and from there, how a certificate program was created.

Notes

30 million developers
300,000 AI practitioners

The goal is to increase it by a factor 10. Train millions of developers to reach billions of people. The mission: Make AI Easy

Book on Amazon: AI and Machine Learning for Coders

TensorFlow.js 101: ML in the Browser and Beyond

Jason Mayes

Discover how to embrace machine learning in JavaScript using TensorFlow.js in the browser and beyond in this speedy talk. Get inspired through a whole bunch of creative prototypes that push the boundaries of what is possible in the modern web browser (things have come a long way) and then take your own first steps with machine learning in minutes. By the end of the talk everyone will understand how to recognize an object of their choice which could then be used in any creative way you can imagine. Familiarity with JavaScript is assumed, but no background in machine learning is required. Come take your first steps with TensorFlow.js!

Day 2 (Nov. 6th, 2020)

Boost Productivity with Keras Ecosystem

Haifeng Jin Keras Team at Google

TensorFlow has built a solid foundation for various machine learning applications, on top of which the Keras ecosystem can really boost the productivity of the developers in building machine learning solutions. Keras has a simple and arbitrarily flexible API for building and training models. However, we still need a lot of manual work to tune the hyperparameters. Fortunately, with Keras Tuner, we can automate the hyperparameter tuning process with minor modifications to the code for building and training the models. To further boost the productivity, we introduce AutoKeras, which fully automates the model building, training, and hyperparameter tuning process. It dramatically reduces the amount of prior knowledge needed of using machine learning for some common tasks. All you need is to define the task and to provide the training data.

Never Have an Unmaintainable Jupyter Notebook Again!

Marco Gorelli

Data visualisation is a fundamental part of Data Science. The talk will start with a practical demonstration (using pandas, scikit-learn, and matplotlib) of how relying on summary statistics and predictions alone can leave you blind to the true nature of your datasets. I will make the point that visualisations are crucial in every step of the Data Science process and therefore that Jupyter Notebooks definitely do belong in Data Science. We will then look at how maintainability is a real challenge for Jupyter Notebooks, especially when trying to keep them under version control with git. Although there exists a plethora of code quality tools for Python scripts (flake8, black, mypy, etc.), most of them don’t work on Jupyter Notebooks. To this end I will present nbQA, which allows any standard Python code quality tool to be run on a Jupyter Notebook. Finally, I will demonstrate how to use it within a workflow which lets practitioners keep the interactivity of their Jupyter Notebooks without having to sacrifice their maintainability.

github
nbdime
nbQA -

Django useful info

Wed, 04 Nov 2020 00:00:00 +0000

Django status: Application that provides an API to check the status of some parts and some utilities like ping.

Startup Resources

Wed, 04 Nov 2020 00:00:00 +0000

How to start a startup

How to Start a Startup: youtube channel. Stanford lectures.
lecture trascripts

Slides

Lecture 1 Sam

Lecture 1 Dustin

People have this vision of being the CEO of a company they started and being on top of the pyramid...
What it's really like: everyone else is your boss -- all of your employees, customers, partners, users, media are you boss.
...

Dave Parker

dkparker.com
Top books for startups
KIC WTIA presentation slides
TAM, SAM, SOM, and LAM: What’s your Launch Addressable Market?

Coursera courses

Coursera Startup Entrepreneurship: Technion Israel Institute of Technology
Coursera Creating and Developing A Tech Startup: offered by École Polytechnique, HEC Paris

Presentations

The single biggest reason why start-ups succeed | Bill Gross:

Founder of IdeaLab
- Idea: ideas are not everything
- Team: execution - adaptability - the customer is the true reality
- Business model
- Funding
- Timing
Top 5 factors in success
1. Timing (42%): AirBnB and Uber came right after the great recession when people needed extra money to overcome their initial rejection of the ideas. CitySearch belongs to the same category when people were looking for cost-effective ways to build web sites. On the other hand, Z.com was an online entertainment company and we had high hopes. But broadband penetration was too low in 1990-2000. A couple of years later when broadband was available, youtube came out perfectly timed and had a great success.
2. Team/Execution (32%)
3. Idea “Truth” Outlier (28%)
4. Business Model (24%)
5. Funding (14%)

Everybody has a plan until they get punched in the face.
  -- Mike Tyson

AI Big World

Maths for AI & ML

Slides

Free Online Books

English

React Resources

Learning React

Seq2Seq + Attention (Minsuk Heo)

Problem

Sequence to sequence

Attention mechanism

Teacher forcing

Stanford: NLP with Machine Learning (2)

Lecture 2: Word Vectors 2 and Word Senses

Lecture Plan

Main idea of word2vec

2. Optimization: Gradient Descent

Gradient Descent - updated equation

Reference

Experimental Creative Writing with the Vectorized Word

Experimental Creative Writing with the Vectorized Word

NLP Events

EMNLP 2020

Business As Mission

Training programs

Neurodivere News and Information

Reference

Markdown Tips and Tricks

How to create the table of contents

subscript and superscript

bold, italics, and bold italics

Math formulae

1. Use mathjax header (preferred)

2. Latex image tags

Jekyll Tips and Tricks

Files and folders

File names

Folder location

Image files

Stanford: NLP with Machine Learning (1)

Lecture 1: Introduction and Word Vectors

Lecture Plan

How do we represent the meaning of a word?

1. As discrete symbols

Representing words as discrete symbols

Representing words by their context

Word vectors

Two model variants

Reference

Interesting ML Projects

130 ML Projects Solved and Explained

Coding Helper

Latin quotes

What is Word Embedding?

What is word embedding?

Word encoding

Word embedding

Embedding over encoding

How to represent words?

Limitations

Basic idea of learning neural network word embeddings

Directly learning low-dimensional word vectors

Main idea of word2vec

Word Similarity

TF Hub Word2Vec

Word2Vec by Jordan Boyd-Graber

Reference

RNN

Text Generation

English

Chinese

Korean

Translation

AI & Deep Learning Concepts

Interesting GAN Projects

GAN

SkyAR

Faceapp

Magic Sand

Face Generation