Jay Vala

Notes 3

Tue, 06 Nov 2018 00:00:00 +0000

Notes 3

Question: What is Stochastic Process?

Answer: A Stochastic Process is one that generates numbers with certain probability distribution (The word Stochastic simply means probablistic or randomly generated, but it is commonly used in case when referring to a sequence of results assumed to be generated by some underlying probability distribution.)

Question: What is Bernoulli trail?

Answer: A random expriment with exactly two results, success or failure.

Question: What is a Probabilty Mass Function?

Answer: Random variables have numeric range hence it is easy to do maths on them rather than doing them on events representing those variables. So we can define a Probability Mass Function of pmf for a random variable which gives out the probability that random variable have different values.

Question: What is Expectation?

Answer: The expectation is the mean of the random variable. So if $X$ is a random variable with pmf $p(x)$ such that $\sum_x|x|p(x) < \infty$, then expectation is

$E(X) = \sum_xx p(x)$

Example: If rolling one die and Y is the value on it face, what is the expectation?

Solution: We have total of 6 possibilties, hence $p(y) =\frac{1}{6}$

$E(Y) = \sum_{1}^{6} y \frac{1}{6}$ $E(Y) = \frac{1}{6}\sum_{1}^{6} y$ $E(Y)= \frac{21}{6} = 3\frac{1}{2}$

Question: What is Varience?

Answer: The varience of a random number is the measure of whether the values if the random variable tends to be consistent over trails or vairy a lot.

In terms of expectation, varience can be found by how much on an average the variable’s value deviate frim the variables expectation level.

$Var(X) = E((X-E(X))^2)$ $Var(X)= E(X)^2 - E^2(X)$

Question: What is Standard Deviation?

Answer: It is the square root of varience. It is denoted by $\sigma$

Example: What is Expectation and Varience of the sum of two unbaised dice rolls? $S = {2,3,…,12}$

Solution: Expectation is given in the formula above, let $E(X)$ be the Expectation of the sum of roll of two dices and $E(Y)$ be the Expectation of roll of one die$.

$E(X) = E(Y) + E(Y)$

which is calculate above,

$E(X) = 3\frac{1}{2} +3\frac{1}{2} = 7$

and Varience becomes

$Var(X) = E((X-E(X))^2)$

As we already know $E(X) = \sum_xx p(x)$ substituting it in above formula we get,

$Var(X) = E((X -\sum_xp(x))^2)$ $Var(X) = \sum_x p(x)(x-\sum_xp(x)^2)$ $Var(X) = \frac{35}{6}$

Notes 2

Mon, 05 Nov 2018 00:00:00 +0000

Bayes Theorem

In conditional probability we have a order of probabilities. That is

$P(A|B) =\frac{P(A\cap B)}{P(B)}$

But it may not happen that we have the probability in this order or may happen that we may not have the probability of $P(B)$, thats when Bayes Theorm comes into play.

Bayes Theorem lets you swap the order of dependence between events. So we can calculate $P(A|B)$ in terms of $P(B|A)$. This is useful as mentioned above when it is difficult to calculate any one of the term in conditinal probability.

So,

$P(B|A) = \frac{P(A\cap B)}{P(A)} = \frac{P(A|B)P(B)}{P(A)}$

Here, the denominator $P(A)$ can be viewed as Normalizing Constant something that ensures we always have a probability function.

So, to evaluate this denominator,

$P(A\cap B) = P(A|B)P(B)$ $P(A|\cap \bar{B}) = P(A| \bar{B})P(\bar{B})$

Therefore, by additivity rule,

$P(A) = P(A\cap B) + P(A\cap \bar{B})$ $P(A)= P(A|B)P(B) + P(A| \bar{B})P(\bar{B})$

Hence, the Bayes Theorem becomes,

$P(B|A) = \frac{P(A|B)P(B)}{P(A|B)P(B) + P(A| \bar{B})P(\bar{B})}$

Example:Treasure chest A contains 20 Gold and 30 Silver coins, Chest B contains 50 Gold and 50 Silver coins. You randomly choose a treasure chest to open, and then randomly choose a coin from that treasure chest. If the coin you choose is gold, then what is the probability that you chose chest A?

Solution:

Chest A = 20 Gold, 30 Silver

Chest B = 50 Gold, 50 Silver

So let $P(A)$ be the probability of selecting Chest A, and $P(B)$ be the probability of selecting Chest B, and let $P(G|A)$ be the probability of selecting a Gold coint from chest A.

$P(A) = \frac{1}{2}$ $P(B) = \frac{1}{2}$ $P(G|A) = \frac{20}{50} = \frac{2}{5}$ $P(G|B) = \frac{50}{100} = \frac{1}{2}$

So, according to Bayes Theorem,Probability of chest being A given that the coin was gold is

$P(A|G) = \frac{P(G|A)P(A)}{P(G)}$

Now, we can clearly see the problem here, we don’t have $P(G)$ but we can express it using the additive method explained above. So,

$P(G) = P(A)P(G|A) + P(\bar{A})P(G|\bar{A})$

But, $\bar{A}$ in this case is $B$, hence the above equation becomes

$P(G) = P(A)P(G|A) + P(B)P(G|B)$

Putting this in above Bayes formula

$P(A|G) = \frac{P(G|A)P(A)}{P(G)}$ $P(A|G) = \frac{P(G|A)P(A)}{P(A)P(G|A) + P(B)P(G|B)}$

Putting the respective values in we get,

$P(A|G) = \frac{\frac{1}{2} . \frac{2}{5}}{\frac{1}{2} . \frac{2}{5} + \frac{1}{2} .\frac{1}{2}}$ $P(A|G) =\frac{\frac{1}{5}}{\frac{1}{5}+\frac{1}{4}}$ $P(A|G) = \frac{1}{5} .\frac{20}{9}$ $P(A|G) = \frac{20}{45}$

Notes 1

Fri, 02 Nov 2018 00:00:00 +0000

Notes: Natural Language Processing and Machine Learning

Question: What are word embedding or what are word vectors?

Answer: Word embedding are learned representation for text where words that have the same meaning have a similar representation. Word embeddings are in fact a class of techniques where individual words are represented as real-valued vectors in a predefined vector space. Each word is mapped to one vector and the vector values are learned in a way that resembles a neural network, and hence the technique is often lumped into the field of deep learning.

Question What is POS tagging?

Answer: Parts of Speech(POS) tagging is a method of assigning Part of Speech such as Noun, Verb, adjectivte etc to each word of the text.

Question: What is TF-IDF matrix?

Answer: Term Frequency Inverse Document Frequency(TF-IDF) is a numerical statistical method that shows how important a word is to a document in a corpus. It contains two terms,

Term Frequecy: It is the measure of how frequently a term appears in a document. Every document is of different length and hence it may happen that a term might appear in several times in a document, so as a measure of normalization we divide the term appearence in document by the length of document.
Inverse Document Frequency: It is the meausre of how important a term is. While calculating Term Frequency every term is given same importance, but it is evedent that terms like “is”, “the” don’t contribute and hence they need to be scaled down and the rare once have to scaledup
- TF = (Number of times term appear in document/Number of document with term in it)
- IDF = log_e(Total Number of documents/ Number of documents with the term in it)

Question: What is Zip’s Law?

Answer: Zip’s Law states that “If we count the frequency of words and rank them according to the frequency of occurance we can explore relationship between the frequency and its frequency in the list known as rank “r”, then Zip’s law says that” $f \propto \frac{1}{r}$

or in other words, there is a constant $k$ such that $f . r= k$

Question: What is a probability distribution function?

Answer: A probability function or probability distribution function distributes a probability mass of 1 throughout given sample space.

Question: What is Uniform Distribution?

Answer: A situation where each outcome is equaly likely is called a Uniform Distribution.

Mathematical Definations:

Conditional Probability: It is updated probability of an event given some prior knowledge. The conditinal probability of an event A given that event B has occured is:

$P(A|B) = \frac{P(A\bigcap B)}{P(B)}$

If $P(B) = 0$ we have

$P(A \bigcap B) = P(B)P(A|B) = P(A)P(B|A)$

Prior Probability: The probability of an event before we consider our additional knowledge.
Posterior Probability: The new probability that results from using of additional knowledge.
Chain Rule:

$P(A_4,A_3,A_2,A_1) = P(A_4|A_3,A_2,A_1).P(A_3|A_2,A_1).P(A_2|A_1).P(A_1)$

Example: Urn 1 has 1 black ball and 2 white balls and Urn 2 has 1 black ball and 3 white balls. Suppose we pick an urn at random and then select a ball from that urnUrn 1 has 1 black ball and 2 white balls and Urn 2 has 1 black ball and 3 white balls. Suppose we pick an urn at random, then what is the probability that we choose a black ball from urn 2

To answer the above question we can use the chain rule.

Let $P(A)$ be the probabilty of picking urn 2, Hence

$P(A) = \frac{1}{2}$

Then Let $P(B|A)$ be the probability of picking black ball from urn 2.

$P(B|A) =\frac{1}{4}$

Hence, using chain rule we can calculate probaility $P(A,B)$

$P(A,B) = P(B|A).P(A)$ $P(A,B) = \frac{1}{4} * \frac{1}{2}$ $P(A,B) = \frac{1}{8}$

Visualizing Model Using Tensorboard

Tue, 23 Oct 2018 00:00:00 +0000

Assignment 2

In this assignment we are going to use Tensorboard a tool provided with tensorflow to visualize and debug(if necessary) our neural networks visually. I am going to visualize all the parameteres that have the properties which enables us to debug our neural networks or gives us a fair intution on how and what goes wrong.

I will try and do this assignment in google colab, although it will be difficult, I will try my best.

For using tensorboard on google colab we need something called we just need to install a single library using pip.

!pip install tensorboardcolab

Requirement already satisfied: tensorboardcolab in /usr/local/lib/python3.6/dist-packages (0.0.19)

Now that we have everything setup and running we have to train a model to get some output on tensorboard

import tensorflow as tf

from tensorboardcolab import TensorBoardColab

Using TensorFlow backend.

# Getting MNIST
# uploading the data
from google.colab import files
files.upload()

Now that the data is loaded, I will create train set and test set from these files and then convert the labels into One-hot encoded ones

import pandas as pd
train_df = pd.read_csv('mnist_train.csv')
test_df = pd.read_csv('mnist_test.csv')

# Seperating labels from the training and testing data
train = train_df.values
trainSet = train[:,1:]
trainLabel = train[:,:1]
test = test_df.values
testSet = test[:,1:]
testLabel = test[:,:1]

# converting labels into one-hot-encoded values
from sklearn.preprocessing import OneHotEncoder
le = OneHotEncoder(handle_unknown='ignore')

le.fit(trainLabel.reshape(-1,1))
trainLabels = le.transform(trainLabel.reshape(-1,1)).toarray()
testLabels = le.transform(testLabel.reshape(-1,1)).toarray()

# Now defining the weights and baises. The input layer have no weight and no baise, 
# and as I am creating a network with two hidden layer I will create weight and bais for hidden layer and output layer
# These weight are 2D arrays and baises are 1D arrays

hidden_1_Weight = tf.Variable(tf.random_uniform([784, 256]),name='weight_hidden_layer1')  # here we need to define what input this layer will be getting and what will be its output(This output is actually the number of hidden units this layer will have)
hidden_2_Weight = tf.Variable(tf.random_uniform([256, 256]), name='weight_hidden_layer2')
outputWeight = tf.Variable(tf.random_uniform([256, 10]), name='weight_output')   # here the input to this layer will be the output of previous layer i.e. 256 and its output will be number of classes it will predict i.e. 10

# biases
hidden_1_Bias = tf.Variable(tf.random_uniform([256]), name='bais_hidden_layer1')
hidden_2_Bias = tf.Variable(tf.random_uniform([256]), name='bais_hidden_layer2')
outputBias = tf.Variable(tf.random_uniform([10]), name='bais_output')

Now as the model weights and baises are completly defined, I will now add summaries to be displayed on tensorboard. So we have different summary options for different things, for example, we have tf.summary.scalar for displaying single values such as loss, accuracy and then we have tf.summary.histogram to visualize matries such as weights and baises . Also we can visualize layer activations and all other variables, but for the scope of this assignment I will only focus on a few of them.

# # Adding summaries
hist_hidden_1 = tf.summary.histogram('hidden_1_weight', hidden_1_Weight)
hist_hidden_2 = tf.summary.histogram('hidden_2_weight', hidden_2_Weight)
output_layer_hist = tf.summary.histogram('Output_Weight', outputWeight)

# PlaceHolders
X = tf.placeholder(tf.float32, [None, 784], name='input_placeholder')
Y = tf.placeholder(tf.float32, [None, 10], name='class_placeholder')

# Lets define the function for multilayer perceptron
def perceptron(x):
     # Hidden fully connected layer with 128 neurons
    layer_1 = tf.nn.relu(tf.add(tf.matmul(x, hidden_1_Weight), hidden_1_Bias))
    # Hidden fully connected layer with 128 neurons
    layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, hidden_2_Weight), hidden_2_Bias))
    # Output fully connected layer with a neuron for each class
    out_layer = tf.matmul(layer_2, outputWeight) + outputBias
    
    return out_layer

# build the network
output = perceptron(X)

For this assignment I am also going to calculate accuracy on each step or epoch just for visualization.

# Evaluate model
correct_pred = tf.equal(tf.argmax(tf.nn.softmax(output), 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))


# adding accuracy scalar to the tensorboard
accuracy_vis = tf.summary.scalar('accu', accuracy)

# Define loss and optimizer
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=output, labels=Y)) 
loss_vis = tf.summary.scalar('loss', loss)
optimizer = tf.train.AdamOptimizer(learning_rate=0.01) 
train_op = optimizer.minimize(loss)

# Initializing the variables
init = tf.global_variables_initializer()

# Merge all summaries into a single operator
merged_summary = tf.summary.merge_all()

batch_size = 100
def batches(data, label,batch_size):
    for i in range(0,data.shape[0],batch_size):
        yield [data[i:i+batch_size], label[i:i+batch_size]]

summary_writer = tf.summary.FileWriter('/content/log',sess.graph)

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    
    total_batches = len(train)//batch_size
    print(total_batches)
    for epoch in range(10):
        avg_cost = 0
      
        # Loop over all batches
        batchedData  = batches(trainSet,trainLabels,batch_size)
        for batch_data in batchedData:
            _,c, summary_,acc = sess.run([train_op, loss, merged_summary, accuracy], feed_dict={X: batch_data[0],Y: batch_data[1]})
            # run summary and accuracy 
            #summary = sess.run([],feed_dict={X: batch_data[0], Y: batch_data[1]})
            
            # Compute average loss
            avg_cost += c / total_batches
            
            # write all the data to the file
            summary_writer.add_summary(summary_, epoch)
        # Display logs per epoch step
        if epoch % 1 == 0:
            print("Epoch:", '%04d' % (epoch+1), "cost={:.9f} accuracy={:.2f}%".format(avg_cost,acc*100 ))
            
            
    print("Optimization Finished!")
    # Test model
    pred = tf.nn.softmax(output)  # Apply softmax to logits
    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(Y, 1))
    # Calculate accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    print("Accuracy:", accuracy.eval({X: testSet, Y: testLabels}))

599
Epoch: 0001 cost=343591.474908259 accuracy=84.85%
Epoch: 0002 cost=1363.470883059 accuracy=89.90%
Epoch: 0003 cost=353.140610641 accuracy=93.94%
Epoch: 0004 cost=145.729113662 accuracy=96.97%
Epoch: 0005 cost=71.671148529 accuracy=98.99%
Epoch: 0006 cost=48.250662688 accuracy=96.97%
Epoch: 0007 cost=34.012532867 accuracy=98.99%
Epoch: 0008 cost=32.737998139 accuracy=95.96%
Epoch: 0009 cost=29.617493640 accuracy=98.99%
Epoch: 0010 cost=28.458745954 accuracy=96.97%
Optimization Finished!
Accuracy: 0.93049306

tbc=TensorBoardColab(graph_path='/content/log')

Wait for 8 seconds...
TensorBoard link:
http://4304f716.ngrok.io

Now you can copy the above link or click here to visit tensorboard page and explore how the training is done.

Multilayer Perceptron

Sat, 13 Oct 2018 00:00:00 +0000

Introduction to Deep Learning

Assignment 1: Multilayer Perceptron MNIST Dataset

NOTE: This Tutorial was done on google colab so it includes some google cloud helper functions.

In this assignment we are going to build a very basic deep learning model called as multilayer perceptron. Also we are going to expriment on it a little bit as suggested on the course website.

Multilayer Perceptron

Multilayer perceptron is a type of feed forward network, it has minimum 3 layers(although we don’t count input and output as layers) input layer, hidden layer and output layer. It uses backpropogation, a technique for superviesed learning, where in it calculates error between given input and the ouput produce by the network and propogates error backwards. This is very interseting concept but it is a topic for some other time. It uses non-linear activation function to introduce non-linearity because most of the real world data is non-linear in nature. You can learn more about them here.

# importing libraries
import tensorflow as tf

# uploading the data to the drive
from google.colab import files
files.upload()

Saving t10k-images-idx3-ubyte.gz to t10k-images-idx3-ubyte.gz
Saving t10k-labels-idx1-ubyte.gz to t10k-labels-idx1-ubyte.gz
Saving train-images-idx3-ubyte.gz to train-images-idx3-ubyte.gz
Saving train-labels-idx1-ubyte.gz to train-labels-idx1-ubyte.gz

Once all the data has been loaded we need to convert this data into some machine readable format, for this on the course website there is a script called conversion.py which is given for this same purpose, I will stick to this method as it is suggested on the course website. However, this is one of the method to obtain the MNIST data, there are other eaiser methods which can be used. For example,

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

This is the most simple one, but you can also download the ready made csv file from Kaggle website here

# Uploading the conversion script
from google.colab import files
files.upload()

 <input type="file" id="files-43a260c5-fcc0-4b5e-8cfc-a08fc2ad3ecf" name="files[]" multiple disabled />
 <output id="result-43a260c5-fcc0-4b5e-8cfc-a08fc2ad3ecf">
  Upload widget is only available when the cell has been executed in the
  current browser session. Please rerun this cell to enable.
  </output>
  <script src="/nbextensions/google.colab/files.js"></script> 


Saving conversions.py to conversions.py

Running this conversion file from here we can use the magic function of jupyter notebook and run them from herem, but before that we need to unpack these files to use them or to be able to covert them using the conversion script. So for that we are going to use gzip library which is inbuilt in the python library module. Lets do that

# Iterating over the files in the folder, checking for '.gz' extinsion and then unpacking them and saving them
import os
import sys
import gzip
import shutil

# Creating a new folder to save all the uncompressed files

dirToSave = os.path.join(os.getcwd(),'data')

try:
    os.stat(dirToSave)
except:
    os.mkdir(dirToSave)
    
for files in os.listdir():
    if files.endswith('.gz'):
        fileName = files.split('.')[0]
        with gzip.open(os.path.join(files), 'rb') as f_input, open(os.path.join(dirToSave,fileName), 'wb') as f_output:
            shutil.copyfileobj(f_input, f_output)

Now that the folder for the data is created I will move the conversion file to the same folder and run the file to do its magic

!ls

conversions.py	t10k-images-idx3-ubyte.gz   train-labels-idx1-ubyte.gz
data		t10k-labels-idx1-ubyte.gz
sample_data	train-images-idx3-ubyte.gz

!mv conversions.py /content/data

! cd data && ls && pwd

conversions.py		t10k-labels-idx1-ubyte	 train-labels-idx1-ubyte
t10k-images-idx3-ubyte	train-images-idx3-ubyte
/content/data

Now lets run the conversion file

!cd data/ && python3 conversions.py -c -n

Converting to .csv...
Training data...
Test data...
...Done.

Converting to .npy...
Training data...
Test data...
...Done.

As the training and testing data csv files are created, lets examin them to see if they are actually there. I do this just to be sure

! cd data/ && ls

conversions.py	     mnist_train.csv	     t10k-labels-idx1-ubyte
mnist_test.csv	     mnist_train_imgs.npy    train-images-idx3-ubyte
mnist_test_imgs.npy  mnist_train_lbls.npy    train-labels-idx1-ubyte
mnist_test_lbls.npy  t10k-images-idx3-ubyte

# Loading the csv files for use
import pandas as pd

train_df = pd.read_csv('/content/data/mnist_train.csv', index_col=False)
test_df = pd.read_csv('/content/data/mnist_test.csv', index_col=False)

train_df.shape

(59999, 785)

test_df.shape

(9999, 785)

train_df.head()

	5	...
0	0	...
1	4	...
2	1	...
3	9	...
4	2	...

5 rows × 785 columns

As we can see that the first column is the class value and we have to seperate it, so lets do that

train = train_df.values
trainSet = train[:,1:]
trainLabel = train[:,:1]
test = test_df.values
testSet = test[:,1:]
testLabel = test[:,:1]

# checking the shape again
print(trainSet.shape, trainLabel.shape, testSet.shape, testLabel.shape)

(59999, 784) (59999, 1) (9999, 784) (9999, 1)

# The labels are not useful as they are categorical hence we need to encode them
from sklearn.preprocessing import OneHotEncoder
import numpy as np
le = OneHotEncoder(handle_unknown='ignore')

le.fit(np.vstack((train[:,:1],testSet[:,:1])))

trainLabels = le.transform(trainLabel).toarray()
testLabels = le.transform(testLabel).toarray()

testLabels[1]

array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0.])

testLabels.shape

(9999, 10)

There are here so lets not worry about this too much and proceed with building the model. The first thing I will do is define my placeholders. So Placeholders, think of them as containers, they will be used to hold data in them. They are variables. So when we create a graph in tensorflow we need to feed them data and we can not do this directly and hence we need some container to hold the data and these containers in tensorflow are called as Placeholder

About the Dataset:

MNIST data set is set of handwritten digits, which are 28 x 28 pixles, in the conversion script, what we have done is we have got the values of intensity of the each pixle. So the total input is 784, hence the shape in the palce holder will be [None, 784], Here the None signifies the amount of data being feed and that will be decided while creating the network, for example: every dataset is of different length and when we define the batch size say in MNIST to 100 than it will create 600 batches(60000 training examples divided by 100) but if the batch size is different, its going to be because of hardware restrictions then the length of last batch might not be the same with every other batch and that makes putting anything in place of None “foolish”, so we dont need to worry about that now. In the second palceholder y we will have shape [None, 10], 10 here represents the number of classes that we are classifing the data into.

Why should one not initialize the weights with 0?

It is really not a good idea to initialize the weights with 0 because all the neurons will calculate the same thing. Do we need it? No because if we would want every neuron to calculate same value than why do we need those deep networks, we can have just one neuron or one layer and do that, we don’t need the deep neural networks.

NOTE: It is not necessary to initialize the baises randomly.

# Placeholders
X = tf.placeholder('float', [None,784]) # Placeholder for inputing the data into the network.
Y = tf.placeholder('float', [None, 10]) # Placeholder for output layer.

# Now defining the weights and baises. The input layer have no weight and no baise, 
# and as I am creating a network with two hidden layer I will create weight and bais for hidden layer and output layer
# These weight are 2D arrays and baises are 1D arrays

hidden_1_Weight = tf.Variable(tf.random_uniform([784, 256]))  # here we need to define what input this layer will be getting and what will be its output(This output is actually the number of hidden units this layer will have)
hidden_2_Weight = tf.Variable(tf.random_uniform([256, 256]))
outputWeight = tf.Variable(tf.random_uniform([256, 10]))   # here the input to this layer will be the output of previous layer i.e. 256 and its output will be number of classes it will predict i.e. 10

# biases
hidden_1_Bias = tf.Variable(tf.random_uniform([256]))
hidden_2_Bias = tf.Variable(tf.random_uniform([256]))
outputBias = tf.Variable(tf.random_uniform([10]))

By Mayranna [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Commons

So in our case the first layer will calculate

$layer 1 = sigmoid((x * h_{1}) + b_{1})$

for layer 2

$layer 2 = sigmoid((x * h_{2}) + b_{2})$

For the ouput layer I must add softmax activation to get the probabilities, but tensorflow has made it easy in such a way that when we calculate the loss we can do that.

If we don’t want to do it for some reason we can add softmax here in the network defination and then just calculate loss in the loss function without applying softmax there a simple cross_entropy from tenosrflow can do that job.

# Lets define the function for multilayer 
def perceptron(x):
     # Hidden fully connected layer with 128 neurons
    layer_1 = tf.nn.relu(tf.add(tf.matmul(x, hidden_1_Weight), hidden_1_Bias))
    # Hidden fully connected layer with 128 neurons
    layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, hidden_2_Weight), hidden_2_Bias))
    # Output fully connected layer with a neuron for each class
    out_layer = tf.matmul(layer_2, outputWeight) + outputBias
    
    return out_layer

# build the network
preceptron_ = perceptron(X)

Loss function minimizes the error between the classification and the original label. I am using here, softmax_cross_entropy_with_logits which is

$cross entropy = - \sum_{1}^{N} y_{o,c} Log(p,c)$

where,

N: total number of classes

y: if the classification is correct, i.e. class label c belongs to the observation o

p: predicted probability of observation o belongs to class c

So, softmax_cross_entropy_with_logits will be

$softmax crossentropy with logits = softmax(- \sum_{1}^{N} y_{o,c} Log(p,c))$

# Define loss and optimizer
loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=preceptron_, labels=Y)) 

optimizer = tf.train.AdamOptimizer(learning_rate=0.001) # I am using AdamOprimizer optimizer as I think it works better in most of the cases, it comes from my personal experience.
train_op = optimizer.minimize(loss_op)
# Initializing the variables
init = tf.global_variables_initializer()

Now as all the variables are initialized we can move to create batches to feed to our network

def batches(data, label,batch_size):
    for i in range(0,data.shape[0],batch_size):
        yield [data[i:i+batch_size], label[i:i+batch_size]]

batch_size = 100

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    total_batches = len(train)//batch_size
    print(total_batches)
    for epoch in range(200):
        avg_cost = 0
        batchedData  = batches(trainSet,trainLabels,batch_size)
        # Loop over all batches
        for batch_data in batchedData:
            trainSetBatch = batch_data[0]
            trainLabelBatch = batch_data[1]
            _, c = sess.run([train_op, loss_op], feed_dict={X: trainSetBatch,Y: trainLabelBatch})
            # Compute average loss
            avg_cost += c / total_batches
            
        # Display logs per epoch step
        if epoch % 1 == 0:
            print("Epoch:", '%04d' % (epoch+1), "cost=", \
                "{:.9f}".format(avg_cost))
            
    print("Optimization Finished!")
    # Test model
    pred = tf.nn.softmax(preceptron_)  # Apply softmax to logits
    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(Y, 1))
    # Calculate accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    print("Accuracy:", accuracy.eval({X: testSet, Y: testLabels}))

599
Epoch: 0001 cost= 233697.866718579
Epoch: 0002 cost= 13862.419193223
Epoch: 0003 cost= 15239.867756075
Epoch: 0004 cost= 14801.515713954
Epoch: 0005 cost= 13218.383375674
Epoch: 0006 cost= 15048.362955593
Epoch: 0007 cost= 15605.303243443
Epoch: 0008 cost= 14244.286502895
Epoch: 0009 cost= 14137.246602818
Epoch: 0010 cost= 13643.962565580
Epoch: 0011 cost= 14308.345506671
Epoch: 0012 cost= 13012.518639057
Epoch: 0013 cost= 12799.497285821
Epoch: 0014 cost= 13468.869916250
Epoch: 0015 cost= 11905.231280484
Epoch: 0016 cost= 11331.125025601
Epoch: 0017 cost= 10646.379243625
Epoch: 0018 cost= 9626.861044366
Epoch: 0019 cost= 9001.553334091
Epoch: 0020 cost= 7626.579372336
Epoch: 0021 cost= 6334.278111014
Epoch: 0022 cost= 5293.664380053
Epoch: 0023 cost= 3738.413927234
Epoch: 0024 cost= 2760.077281130
Epoch: 0025 cost= 2006.898739831
Epoch: 0026 cost= 1393.388739605
Epoch: 0027 cost= 1061.398122862
Epoch: 0028 cost= 720.131391260
Epoch: 0029 cost= 528.895305433
Epoch: 0030 cost= 371.910963326
Epoch: 0031 cost= 289.990093832
Epoch: 0032 cost= 211.158631602
Epoch: 0033 cost= 141.923613281
Epoch: 0034 cost= 112.115534271
Epoch: 0035 cost= 90.701224615
Epoch: 0036 cost= 94.358168000
Epoch: 0037 cost= 61.984902694
Epoch: 0038 cost= 60.687041363
Epoch: 0039 cost= 42.428540849
Epoch: 0040 cost= 45.945703943
Epoch: 0041 cost= 35.301657116
Epoch: 0042 cost= 33.531888366
Epoch: 0043 cost= 24.714800139
Epoch: 0044 cost= 26.019582580
Epoch: 0045 cost= 17.728049701
Epoch: 0046 cost= 17.548398735
Epoch: 0047 cost= 39.107907718
Epoch: 0048 cost= 21.521840305
Epoch: 0049 cost= 8.725962788
Epoch: 0050 cost= 9.524569323
Epoch: 0051 cost= 5.639854976
Epoch: 0052 cost= 13.697699557
Epoch: 0053 cost= 28.470761760
Epoch: 0054 cost= 11.467136455
Epoch: 0055 cost= 10.369889293
Epoch: 0056 cost= 7.239818920
Epoch: 0057 cost= 11.569948143
Epoch: 0058 cost= 11.443468886
Epoch: 0059 cost= 8.718404733
Epoch: 0060 cost= 13.267106948
Epoch: 0061 cost= 9.522318864
Epoch: 0062 cost= 7.704388247
Epoch: 0063 cost= 6.505781692
Epoch: 0064 cost= 4.287839644
Epoch: 0065 cost= 4.067495077
Epoch: 0066 cost= 5.511625610
Epoch: 0067 cost= 9.277953480
Epoch: 0068 cost= 5.771217371
Epoch: 0069 cost= 7.822138380
Epoch: 0070 cost= 5.067580908
Epoch: 0071 cost= 3.542238488
Epoch: 0072 cost= 2.314385025
Epoch: 0073 cost= 4.829883155
Epoch: 0074 cost= 6.732986233
Epoch: 0075 cost= 4.147152578
Epoch: 0076 cost= 4.814162664
Epoch: 0077 cost= 4.195318871
Epoch: 0078 cost= 4.816954190
Epoch: 0079 cost= 6.405923215
Epoch: 0080 cost= 3.548350426
Epoch: 0081 cost= 6.843513339
Epoch: 0082 cost= 3.730189537
Epoch: 0083 cost= 2.395861579
Epoch: 0084 cost= 1.291224121
Epoch: 0085 cost= 2.609014182
Epoch: 0086 cost= 8.136346506
Epoch: 0087 cost= 6.911538018
Epoch: 0088 cost= 1.613900936
Epoch: 0089 cost= 1.792066425
Epoch: 0090 cost= 1.509146715
Epoch: 0091 cost= 2.816762689
Epoch: 0092 cost= 5.840670672
Epoch: 0093 cost= 4.645757224
Epoch: 0094 cost= 2.435126027
Epoch: 0095 cost= 8.339016823
Epoch: 0096 cost= 1.770155966
Epoch: 0097 cost= 1.422567249
Epoch: 0098 cost= 1.212486510
Epoch: 0099 cost= 1.109428637
Epoch: 0100 cost= 3.898369765
Epoch: 0101 cost= 3.376135472
Epoch: 0102 cost= 4.315077318
Epoch: 0103 cost= 4.274662701
Epoch: 0104 cost= 3.012072559
Epoch: 0105 cost= 1.683379329
Epoch: 0106 cost= 1.264098068
Epoch: 0107 cost= 1.163033118
Epoch: 0108 cost= 1.178344884
Epoch: 0109 cost= 3.371009719
Epoch: 0110 cost= 2.942797631
Epoch: 0111 cost= 1.653774604
Epoch: 0112 cost= 2.065821347
Epoch: 0113 cost= 9.031573909
Epoch: 0114 cost= 2.762784667
Epoch: 0115 cost= 3.431726249
Epoch: 0116 cost= 1.516621503
Epoch: 0117 cost= 1.402665269
Epoch: 0118 cost= 3.369300493
Epoch: 0119 cost= 1.436823709
Epoch: 0120 cost= 0.989465624
Epoch: 0121 cost= 1.465438136
Epoch: 0122 cost= 2.069973967
Epoch: 0123 cost= 1.271443909
Epoch: 0124 cost= 1.228639121
Epoch: 0125 cost= 3.035652603
Epoch: 0126 cost= 4.367832982
Epoch: 0127 cost= 1.883182552
Epoch: 0128 cost= 0.988018134
Epoch: 0129 cost= 1.219982436
Epoch: 0130 cost= 2.460366394
Epoch: 0131 cost= 1.444995319
Epoch: 0132 cost= 0.796145169
Epoch: 0133 cost= 1.866650649
Epoch: 0134 cost= 2.729383657
Epoch: 0135 cost= 1.498433346
Epoch: 0136 cost= 0.894979498
Epoch: 0137 cost= 0.685542778
Epoch: 0138 cost= 1.086774313
Epoch: 0139 cost= 1.573417341
Epoch: 0140 cost= 1.511079420
Epoch: 0141 cost= 3.592942662
Epoch: 0142 cost= 1.756971295
Epoch: 0143 cost= 2.371372009
Epoch: 0144 cost= 1.350900717
Epoch: 0145 cost= 3.918655687
Epoch: 0146 cost= 1.380350943
Epoch: 0147 cost= 1.348579527
Epoch: 0148 cost= 1.050694610
Epoch: 0149 cost= 0.912132167
Epoch: 0150 cost= 0.782553376
Epoch: 0151 cost= 0.597382387
Epoch: 0152 cost= 0.594046318
Epoch: 0153 cost= 0.592238037
Epoch: 0154 cost= 0.592521275
Epoch: 0155 cost= 0.594104953
Epoch: 0156 cost= 0.589922921
Epoch: 0157 cost= 0.703515567
Epoch: 0158 cost= 4.097817484
Epoch: 0159 cost= 3.709944400
Epoch: 0160 cost= 3.080095094
Epoch: 0161 cost= 1.277377238
Epoch: 0162 cost= 0.838328730
Epoch: 0163 cost= 0.941523782
Epoch: 0164 cost= 1.633839919
Epoch: 0165 cost= 1.591967852
Epoch: 0166 cost= 0.826101982
Epoch: 0167 cost= 0.960754743
Epoch: 0168 cost= 0.991728673
Epoch: 0169 cost= 1.230600796
Epoch: 0170 cost= 1.051815032
Epoch: 0171 cost= 1.671720224
Epoch: 0172 cost= 1.550530605
Epoch: 0173 cost= 0.948061457
Epoch: 0174 cost= 0.775590160
Epoch: 0175 cost= 0.687840205
Epoch: 0176 cost= 0.688999365
Epoch: 0177 cost= 1.429095754
Epoch: 0178 cost= 3.105333766
Epoch: 0179 cost= 1.452365677
Epoch: 0180 cost= 1.176313281
Epoch: 0181 cost= 0.766753364
Epoch: 0182 cost= 3.400296347
Epoch: 0183 cost= 1.933165025
Epoch: 0184 cost= 1.105114213
Epoch: 0185 cost= 1.771966526
Epoch: 0186 cost= 2.085469130
Epoch: 0187 cost= 0.894252141
Epoch: 0188 cost= 1.149476664
Epoch: 0189 cost= 0.906636394
Epoch: 0190 cost= 1.220944617
Epoch: 0191 cost= 0.875083064
Epoch: 0192 cost= 1.119831198
Epoch: 0193 cost= 0.861701711
Epoch: 0194 cost= 0.755726190
Epoch: 0195 cost= 1.848937889
Epoch: 0196 cost= 1.442416712
Epoch: 0197 cost= 5.017063204
Epoch: 0198 cost= 0.906185662
Epoch: 0199 cost= 1.311479466
Epoch: 0200 cost= 0.750225017
Optimization Finished!
Accuracy: 0.7509751

Finding K in K-Means

Mon, 09 Jul 2018 00:00:00 +0000

Finding K in K-Means

So when applying K-Means Clustring, one comes at a point where he/she has to decide how many cluster they want, now one can not go and tell 2,3, or 4 clusters there should be a some evidence that clustering the data into ‘k’ clusters will yeild good results. So I met this problem, after searching for what can be done about this problem I stumble upon a something called ‘Silhouette Analysis’

Silhouette Analysis

Silhouette Analysis takes into considration of how well a particular data point lies within a given cluster. So lets take an example to understand how this works, Say we are using K-Means Clustring for ‘k’ number of clusters. Now for a data point ‘i’, we define average distance from all points in the same cluster as avg(i) = a(i), so we can interpret a(i) as measure of how well that point belongs in the cluster or to put it simply how well that point belongs in that cluster, so smaller the value, better the assignment of that data point to that cluster. Similarly, lets say b(i) is the average dissimlarity, that means b(i) the lowest average distance of data point i to all points in any other cluster, of which i is not a member of. The cluster with this lowest average dissimilarity is said to be the “neighbouring cluster” of i because it is the next best fit cluster for point i

from IPython.display import Math
Math(r's(i) = \frac{b(i)-a(i)}{max(b(i),a(i))}')

$s(i) = \frac{b(i)-a(i)}{max(b(i),a(i))}$

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import numpy as np
from sklearn.metrics import silhouette_samples, silhouette_score

# Getting all the english and german data

import os, re

label = []
content = []

for root, dirs, files in os.walk('/home/jay/Data/English/'):
    for file in files:
        topic = root.split(os.path.sep)[-2]
        if file.endswith('.txt') and file != 'log.txt':
            with open(os.path.join(root, file)) as f:
                content.append(f.read())
                label.append(topic)
                

        
for root, dirs, files in os.walk('/home/jay/Data/German/'):
    for file in files:
        topic = root.split(os.path.sep)[-2]
        if file.endswith('.txt') and file != 'log.txt':
            with open(os.path.join(root, file)) as i:
                content.append(i.read())
                label.append(topic)

# running the k-means for 32 clusters
# creating tf-idf matrix

tfidf_matrix_constructor = TfidfVectorizer(max_df=0.8, max_features=200000, min_df=0.2, ngram_range=(1,3))

tfidf_matrix = tfidf_matrix_constructor.fit_transform(content)

# printing the shape of the resultant matrix to ensure I am right
print(tfidf_matrix.shape)

(9986, 59)

# now applying k-means for range 1-32 and calculating silhouette_score for each cluster

for k in range(2 ,33):
    
    # define k-means constructor
    kmeans = KMeans(n_clusters=k,random_state=10)
    
    cluster_labels = kmeans.fit_predict(tfidf_matrix.toarray())
    
    # Calculating silhouette_score for k
    
    score = silhouette_score(tfidf_matrix.toarray(), cluster_labels, random_state=10)
    print("The silhouette score for {} clusters is {}".format(k,score))

The silhouette score for 2 clusters is 0.2092956601357311
The silhouette score for 3 clusters is 0.1463292693620873
The silhouette score for 4 clusters is 0.14046381676564526
The silhouette score for 5 clusters is 0.14694926636316893
The silhouette score for 6 clusters is 0.14478803184773398
The silhouette score for 7 clusters is 0.17189746569523418
The silhouette score for 8 clusters is 0.17280711786498046
The silhouette score for 9 clusters is 0.18248360175164505
The silhouette score for 10 clusters is 0.185809463489821
The silhouette score for 11 clusters is 0.19185632542163303
The silhouette score for 12 clusters is 0.19569188868141868
The silhouette score for 13 clusters is 0.19476772871615036
The silhouette score for 14 clusters is 0.20251183229887623
The silhouette score for 15 clusters is 0.19103405902025516
The silhouette score for 16 clusters is 0.19483617751124183
The silhouette score for 17 clusters is 0.19140067934731583
The silhouette score for 18 clusters is 0.19934328007322114
The silhouette score for 19 clusters is 0.19446860597424745
The silhouette score for 20 clusters is 0.19904443285585602
The silhouette score for 21 clusters is 0.195384733891542
The silhouette score for 22 clusters is 0.19990294699010308
The silhouette score for 23 clusters is 0.20434103038355314
The silhouette score for 24 clusters is 0.20162229208943913
The silhouette score for 25 clusters is 0.20001554425267193
The silhouette score for 26 clusters is 0.20297321222570094
The silhouette score for 27 clusters is 0.2023863657790054
The silhouette score for 28 clusters is 0.19960601038803405
The silhouette score for 29 clusters is 0.1985121190442007
The silhouette score for 30 clusters is 0.20256036983356498
The silhouette score for 31 clusters is 0.2001042315801357
The silhouette score for 32 clusters is 0.20251412402591076

So the higher the value of Silhouette Score, better the number of clusters.

Visulaizing LDA results

Mon, 28 May 2018 00:00:00 +0000

This post will not work here so please click here

If you have trouble viewing it please contact me at: jay.vala@msn.com

Latent Dirichlet Allocation(Topic Modelling)-BaseLine

Sun, 27 May 2018 00:00:00 +0000

Latent Dirichlet Allocation(LDA)

Topic modeling can be useful when having a large corpus, when we want to unearth the meaning or categories of the data we have, which is too large to be done manually. In simple terms LDA is probabilistic unsupervised models that gives out top topics. So suppose we have a set of documents. we’ve chosen some fixed number of K topics to discover, and want to use LDA to learn the topic representation of each document and the words associated to each topic. LDA uses collapsed Gibbs sampling. How?

Go through each document, and randomly assign each word(w) in the document(d) to one of the K topics(t).
This random assignment already gives us topic assignment and the topic distribution but they won’t be good at all.
So to importve upon them we have to,
- Go through every word in the document and compute two things
  - p(topic t | document d), i.e. the words(w) in document d currently assgin to topic t.
  - p(word w | topic t), i.e. the assignment of topic over all the document because of this word(w).
  - This a generative model and hence we have to reassign a new topic to a word and repeat.

To learn about it more and to get an intution behind the idea how LDA works click here I got the intusion from there.

Steps

Pre-processing and training corpus creation
Building dictionary
Feature extraction
LDA model training

Pre-processing text for LDA is a little bit different for LDA than what I did for RNN screated in this post.

# Lets pre-process them.
import os
import random
from nltk.corpus import stopwords
import string
import sys
import spacy
import re
import logging
from gensim.models.ldamodel import LdaModel as Lda
from gensim import corpora

# NLP model from spacy
nlp = spacy.load('en')

Pre-processing Text

def preprocess(text):
    """Returns text after preprocessing
    :params:list of text
    :returns:list of text after manipulation"""
    
    pun = string.punctuation+'$€¥₹£|–'
    regex = re.compile('[%s]' % re.escape(string.punctuation))
    punct = " ".join([i for i in text.lower().split() if i.split() not in pun.split()])
    stop = " ".join([i for i in punct.lower().split() if i not in stopwords.words('english')])
    digit = re.sub(r'\d+','',stop)
    # removing punct again
    punct_ = regex.sub('',digit)
    doc = nlp(punct_)
    lam = " ".join(word.lemma_ for word in doc)
    x = lam.split()
    y = [s for s in x if len(s) > 2]
    return y

# Lets load text from every doc into a list 

document_list = []

for root, dirs, files in os.walk('/home/jay/Thesis_1/Data/Data_EN'):
    for file in files:
        if file != 'log.txt':
            with open(os.path.join(root, file), 'r') as f:
                document_list.append(f.read())

# Lets divide the documents into test and train set
# I am taking 20% of documents for test set, but before that lets just suffle it.
random.shuffle(document_list)
train = document_list[round(len(document_list)*.2):]
test = document_list[:round(len(document_list)*.2)]

len(train)

# preprocess the training set
cleaned = [preprocess(doc) for doc in train]

Dictionary Building

For dictionary building gensim requires all the words in corpus. So lets create a list of words in dictionary

# For building dictionary I will use gensim.
# Dictionary are nothing but every unique term with its unique id as we have already created for training RNNs.
# We can also create 'hashdictionary' were it uses hashing algorithm which will increase speed, but I will not worry
# about it as my corpus is small

dictionary = corpora.Dictionary(cleaned)

print(dictionary)

Dictionary(19634 unique tokens: ['-PRON-', 'access', 'achieve', 'achievement', 'act']...)

Now that dictonary is created we need to filter out the dictonary, we will filter out the words that occur in less than 4 document and words that occur in more than 40% of the documents. We do this because these words do not contibute in the different themes and topics that are in the corpus.

# removing extremes
dictionary.filter_extremes(no_below=4, no_above=0.4)

print(dictionary)

Dictionary(8231 unique tokens: ['-PRON-', 'access', 'achieve', 'achievement', 'adapt']...)

Feature Extraction

Now that dictionary is created, moving on to the next step, extracting features. Gensim provides use to necessary tools to extract features out of the courpus. Feature extraction is nothing but generating but the frequencies of all the words in the vocabulary for that particular word.

doc_term_matrix = [dictionary.doc2bow(doc) for doc in cleaned]

Model Building

# As I know that I have only 32 topics in the corpus, I will set the num_topic argument as 32
# To see the progress I added loggig as suggested in Gensim Tutorial 
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

ldamodel = Lda(doc_term_matrix, num_topics=32, id2word = dictionary, passes=50, iterations=500)

2018-05-28 18:52:24,623 : INFO : using symmetric alpha at 0.03125
2018-05-28 18:52:24,627 : INFO : using symmetric eta at 0.03125
2018-05-28 18:52:24,631 : INFO : using serial LDA version on this node
2018-05-28 18:52:24,696 : INFO : running online (multi-pass) LDA training, 32 topics, 50 passes over the supplied corpus of 3692 documents, updating model once every 2000 documents, evaluating perplexity every 3692 documents, iterating 500x with a convergence threshold of 0.001000
2018-05-28 18:52:24,698 : INFO : PROGRESS: pass 0, at document #2000/3692
2018-05-28 18:53:04,688 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:53:04,760 : INFO : topic #14 (0.031): 0.013*"research" + 0.010*"innovation" + 0.009*"sector" + 0.007*"protection" + 0.007*"woman" + 0.006*"europe" + 0.006*"medium" + 0.005*"content" + 0.005*"financial" + 0.005*"interactive"
2018-05-28 18:53:04,763 : INFO : topic #1 (0.031): 0.007*"person" + 0.007*"directive" + 0.006*"product" + 0.006*"right" + 0.005*"article" + 0.005*"request" + 0.005*"network" + 0.004*"law" + 0.004*"mercury" + 0.004*"agency"
2018-05-28 18:53:04,764 : INFO : topic #29 (0.031): 0.009*"research" + 0.007*"human" + 0.007*"right" + 0.007*"health" + 0.006*"directive" + 0.005*"employment" + 0.005*"animal" + 0.005*"law" + 0.004*"woman" + 0.004*"committee"
2018-05-28 18:53:04,766 : INFO : topic #22 (0.031): 0.013*"financial" + 0.010*"fund" + 0.009*"education" + 0.006*"directive" + 0.005*"strategy" + 0.005*"training" + 0.004*"instrument" + 0.004*"management" + 0.004*"nuclear" + 0.004*"safety"
2018-05-28 18:53:04,768 : INFO : topic #3 (0.031): 0.023*"passenger" + 0.014*"tax" + 0.013*"directive" + 0.010*"right" + 0.008*"air" + 0.007*"car" + 0.007*"travel" + 0.007*"carrier" + 0.006*"liability" + 0.005*"water"
2018-05-28 18:53:04,769 : INFO : topic diff=18.296263, rho=1.000000
2018-05-28 18:53:22,841 : INFO : -7.718 per-word bound, 210.6 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 18:53:22,843 : INFO : PROGRESS: pass 0, at document #3692/3692
2018-05-28 18:53:35,697 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 18:53:35,757 : INFO : topic #11 (0.031): 0.015*"custom" + 0.014*"security" + 0.009*"directive" + 0.009*"control" + 0.007*"law" + 0.006*"trade" + 0.006*"document" + 0.005*"issue" + 0.005*"management" + 0.005*"common"
2018-05-28 18:53:35,758 : INFO : topic #10 (0.031): 0.017*"euro" + 0.011*"capital" + 0.009*"financial" + 0.008*"directive" + 0.007*"right" + 0.007*"pension" + 0.007*"coin" + 0.006*"company" + 0.006*"bank" + 0.006*"care"
2018-05-28 18:53:35,760 : INFO : topic #1 (0.031): 0.013*"person" + 0.008*"directive" + 0.007*"citizen" + 0.007*"network" + 0.006*"right" + 0.006*"sis" + 0.006*"contract" + 0.006*"request" + 0.006*"drug" + 0.005*"product"
2018-05-28 18:53:35,762 : INFO : topic #18 (0.031): 0.010*"directive" + 0.007*"application" + 0.007*"minor" + 0.007*"condition" + 0.006*"applicant" + 0.006*"air" + 0.006*"network" + 0.005*"lay" + 0.005*"import" + 0.005*"common"
2018-05-28 18:53:35,764 : INFO : topic #26 (0.031): 0.015*"treaty" + 0.011*"financial" + 0.007*"procedure" + 0.006*"progress" + 0.006*"acquis" + 0.006*"security" + 0.005*"court" + 0.004*"agreement" + 0.004*"rate" + 0.004*"government"
2018-05-28 18:53:35,765 : INFO : topic diff=0.868584, rho=0.707107
2018-05-28 18:53:35,767 : INFO : PROGRESS: pass 1, at document #2000/3692
2018-05-28 18:53:47,128 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:53:47,218 : INFO : topic #28 (0.031): 0.026*"right" + 0.016*"datum" + 0.009*"protection" + 0.009*"judicial" + 0.008*"justice" + 0.007*"directive" + 0.007*"treaty" + 0.006*"security" + 0.006*"fundamental" + 0.006*"matter"
2018-05-28 18:53:47,222 : INFO : topic #24 (0.031): 0.020*"general" + 0.018*"interest" + 0.015*"paper" + 0.013*"research" + 0.013*"medium" + 0.010*"green" + 0.008*"consultation" + 0.008*"sport" + 0.007*"organisation" + 0.007*"society"
2018-05-28 18:53:47,225 : INFO : topic #11 (0.031): 0.018*"custom" + 0.015*"security" + 0.010*"control" + 0.008*"directive" + 0.007*"trade" + 0.007*"document" + 0.007*"internal" + 0.006*"law" + 0.006*"management" + 0.006*"issue"
2018-05-28 18:53:47,229 : INFO : topic #5 (0.031): 0.019*"transport" + 0.016*"regional" + 0.015*"region" + 0.015*"vehicle" + 0.011*"fund" + 0.010*"road" + 0.010*"partnership" + 0.009*"initiative" + 0.008*"sea" + 0.008*"integration"
2018-05-28 18:53:47,231 : INFO : topic #26 (0.031): 0.015*"treaty" + 0.012*"financial" + 0.007*"procedure" + 0.006*"budgetary" + 0.006*"deficit" + 0.006*"progress" + 0.006*"rate" + 0.006*"court" + 0.005*"acquis" + 0.005*"government"
2018-05-28 18:53:47,233 : INFO : topic diff=0.585917, rho=0.509912
2018-05-28 18:54:00,950 : INFO : -7.385 per-word bound, 167.2 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 18:54:00,951 : INFO : PROGRESS: pass 1, at document #3692/3692
2018-05-28 18:54:10,696 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 18:54:10,778 : INFO : topic #31 (0.031): 0.020*"progress" + 0.018*"accession" + 0.016*"candidate" + 0.013*"acquis" + 0.010*"environment" + 0.009*"turkey" + 0.009*"sec" + 0.009*"partnership" + 0.009*"priority" + 0.008*"negotiation"
2018-05-28 18:54:10,780 : INFO : topic #3 (0.031): 0.024*"directive" + 0.024*"passenger" + 0.023*"air" + 0.021*"transport" + 0.021*"tax" + 0.012*"carrier" + 0.011*"vehicle" + 0.010*"vat" + 0.009*"emission" + 0.009*"rail"
2018-05-28 18:54:10,781 : INFO : topic #18 (0.031): 0.015*"directive" + 0.010*"minor" + 0.010*"rail" + 0.009*"application" + 0.008*"condition" + 0.008*"interoperability" + 0.008*"network" + 0.008*"standard" + 0.007*"lay" + 0.007*"agency"
2018-05-28 18:54:10,784 : INFO : topic #30 (0.031): 0.019*"security" + 0.012*"border" + 0.010*"external" + 0.008*"migration" + 0.007*"defence" + 0.007*"strategy" + 0.006*"relation" + 0.006*"management" + 0.006*"crisis" + 0.006*"agreement"
2018-05-28 18:54:10,785 : INFO : topic #29 (0.031): 0.017*"research" + 0.014*"woman" + 0.013*"human" + 0.012*"right" + 0.011*"child" + 0.010*"health" + 0.008*"man" + 0.007*"equality" + 0.006*"scientific" + 0.006*"protection"
2018-05-28 18:54:10,787 : INFO : topic diff=0.722667, rho=0.509912
2018-05-28 18:54:10,790 : INFO : PROGRESS: pass 2, at document #2000/3692
2018-05-28 18:54:21,708 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:54:21,815 : INFO : topic #27 (0.031): 0.018*"datum" + 0.010*"access" + 0.009*"protection" + 0.007*"internet" + 0.006*"plan" + 0.006*"partner" + 0.006*"regional" + 0.006*"network" + 0.006*"transport" + 0.005*"asylum"
2018-05-28 18:54:21,818 : INFO : topic #3 (0.031): 0.026*"transport" + 0.024*"air" + 0.023*"directive" + 0.023*"passenger" + 0.020*"tax" + 0.012*"vehicle" + 0.012*"carrier" + 0.010*"rail" + 0.009*"vat" + 0.009*"emission"
2018-05-28 18:54:21,819 : INFO : topic #31 (0.031): 0.022*"progress" + 0.019*"accession" + 0.017*"candidate" + 0.016*"acquis" + 0.011*"turkey" + 0.010*"sec" + 0.010*"negotiation" + 0.010*"environment" + 0.009*"enlargement" + 0.009*"capacity"
2018-05-28 18:54:21,820 : INFO : topic #14 (0.031): 0.025*"research" + 0.018*"people" + 0.018*"innovation" + 0.017*"young" + 0.016*"youth" + 0.014*"cultural" + 0.011*"europe" + 0.009*"sector" + 0.008*"culture" + 0.008*"field"
2018-05-28 18:54:21,822 : INFO : topic #18 (0.031): 0.017*"directive" + 0.013*"rail" + 0.011*"interoperability" + 0.010*"network" + 0.010*"minor" + 0.010*"agency" + 0.009*"equipment" + 0.008*"application" + 0.008*"technical" + 0.008*"condition"
2018-05-28 18:54:21,823 : INFO : topic diff=0.694411, rho=0.454264
2018-05-28 18:54:34,992 : INFO : -7.278 per-word bound, 155.2 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 18:54:34,993 : INFO : PROGRESS: pass 2, at document #3692/3692
2018-05-28 18:54:43,851 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 18:54:43,920 : INFO : topic #20 (0.031): 0.022*"trade" + 0.011*"agreement" + 0.011*"consumer" + 0.010*"committee" + 0.009*"standard" + 0.008*"sector" + 0.007*"product" + 0.007*"business" + 0.007*"legislation" + 0.006*"directive"
2018-05-28 18:54:43,924 : INFO : topic #27 (0.031): 0.020*"datum" + 0.011*"access" + 0.010*"protection" + 0.007*"plan" + 0.007*"internet" + 0.006*"network" + 0.006*"partner" + 0.006*"region" + 0.006*"regional" + 0.006*"instrument"
2018-05-28 18:54:43,928 : INFO : topic #9 (0.031): 0.018*"financial" + 0.012*"crime" + 0.011*"terrorist" + 0.009*"fraud" + 0.009*"prevention" + 0.009*"infrastructure" + 0.009*"combat" + 0.008*"noise" + 0.008*"terrorism" + 0.008*"money"
2018-05-28 18:54:43,931 : INFO : topic #3 (0.031): 0.028*"transport" + 0.026*"air" + 0.023*"directive" + 0.022*"passenger" + 0.019*"tax" + 0.015*"vehicle" + 0.012*"carrier" + 0.011*"vat" + 0.011*"duty" + 0.009*"emission"
2018-05-28 18:54:43,937 : INFO : topic #13 (0.031): 0.041*"directive" + 0.019*"law" + 0.014*"court" + 0.013*"right" + 0.010*"person" + 0.010*"legal" + 0.009*"proceeding" + 0.008*"eec" + 0.008*"worker" + 0.008*"case"
2018-05-28 18:54:43,940 : INFO : topic diff=0.752502, rho=0.454264
2018-05-28 18:54:43,942 : INFO : PROGRESS: pass 3, at document #2000/3692
2018-05-28 18:54:54,499 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:54:54,594 : INFO : topic #18 (0.031): 0.020*"directive" + 0.016*"rail" + 0.013*"interoperability" + 0.013*"agency" + 0.012*"equipment" + 0.012*"network" + 0.011*"technical" + 0.010*"minor" + 0.009*"railway" + 0.009*"condition"
2018-05-28 18:54:54,595 : INFO : topic #12 (0.031): 0.027*"fishing" + 0.017*"fishery" + 0.012*"committee" + 0.011*"vessel" + 0.010*"ecb" + 0.010*"bank" + 0.009*"humanitarian" + 0.009*"central" + 0.008*"group" + 0.007*"financial"
2018-05-28 18:54:54,597 : INFO : topic #1 (0.031): 0.018*"person" + 0.012*"contract" + 0.010*"citizen" + 0.010*"drug" + 0.009*"condition" + 0.008*"request" + 0.008*"network" + 0.008*"entry" + 0.007*"residence" + 0.007*"right"
2018-05-28 18:54:54,598 : INFO : topic #4 (0.031): 0.019*"right" + 0.015*"agreement" + 0.014*"europol" + 0.013*"online" + 0.012*"property" + 0.012*"patent" + 0.012*"protection" + 0.011*"intellectual" + 0.010*"access" + 0.010*"enforcement"
2018-05-28 18:54:54,600 : INFO : topic #15 (0.031): 0.057*"directive" + 0.031*"safety" + 0.021*"health" + 0.018*"risk" + 0.014*"worker" + 0.013*"eec" + 0.010*"exposure" + 0.010*"requirement" + 0.010*"protection" + 0.009*"convention"
2018-05-28 18:54:54,602 : INFO : topic diff=0.675248, rho=0.413591
2018-05-28 18:55:07,337 : INFO : -7.219 per-word bound, 149.0 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 18:55:07,339 : INFO : PROGRESS: pass 3, at document #3692/3692
2018-05-28 18:55:15,041 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 18:55:15,124 : INFO : topic #11 (0.031): 0.027*"security" + 0.021*"custom" + 0.015*"control" + 0.010*"export" + 0.009*"document" + 0.009*"internal" + 0.008*"common" + 0.007*"trade" + 0.007*"management" + 0.007*"issue"
2018-05-28 18:55:15,127 : INFO : topic #6 (0.031): 0.037*"product" + 0.029*"food" + 0.020*"consumer" + 0.014*"agency" + 0.014*"health" + 0.007*"foodstuff" + 0.007*"name" + 0.007*"application" + 0.007*"modify" + 0.006*"agricultural"
2018-05-28 18:55:15,129 : INFO : topic #20 (0.031): 0.026*"trade" + 0.013*"consumer" + 0.011*"agreement" + 0.010*"committee" + 0.010*"standard" + 0.008*"sector" + 0.008*"business" + 0.007*"legislation" + 0.007*"product" + 0.006*"access"
2018-05-28 18:55:15,131 : INFO : topic #3 (0.031): 0.033*"transport" + 0.027*"air" + 0.022*"passenger" + 0.022*"directive" + 0.018*"vehicle" + 0.017*"tax" + 0.012*"carrier" + 0.011*"duty" + 0.011*"vat" + 0.010*"airport"
2018-05-28 18:55:15,133 : INFO : topic #25 (0.031): 0.073*"aid" + 0.023*"competition" + 0.018*"article" + 0.012*"treaty" + 0.012*"grant" + 0.009*"agreement" + 0.009*"guideline" + 0.009*"sector" + 0.008*"application" + 0.008*"investment"
2018-05-28 18:55:15,134 : INFO : topic diff=0.667393, rho=0.413591
2018-05-28 18:55:15,136 : INFO : PROGRESS: pass 4, at document #2000/3692
2018-05-28 18:55:23,624 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:55:23,720 : INFO : topic #22 (0.031): 0.030*"fund" + 0.022*"financial" + 0.017*"education" + 0.016*"training" + 0.015*"project" + 0.011*"eur" + 0.011*"assistance" + 0.011*"period" + 0.011*"million" + 0.010*"instrument"
2018-05-28 18:55:23,723 : INFO : topic #3 (0.031): 0.035*"transport" + 0.028*"air" + 0.022*"passenger" + 0.021*"directive" + 0.019*"vehicle" + 0.016*"tax" + 0.012*"carrier" + 0.010*"airport" + 0.010*"duty" + 0.009*"vat"
2018-05-28 18:55:23,726 : INFO : topic #9 (0.031): 0.020*"financial" + 0.017*"crime" + 0.012*"terrorist" + 0.011*"prevention" + 0.010*"combat" + 0.010*"fraud" + 0.010*"terrorism" + 0.009*"infrastructure" + 0.009*"noise" + 0.008*"fight"
2018-05-28 18:55:23,731 : INFO : topic #23 (0.031): 0.036*"convention" + 0.035*"maritime" + 0.031*"ship" + 0.019*"sea" + 0.018*"port" + 0.017*"pollution" + 0.016*"vessel" + 0.015*"marine" + 0.012*"party" + 0.012*"protocol"
2018-05-28 18:55:23,735 : INFO : topic #1 (0.031): 0.019*"person" + 0.013*"contract" + 0.011*"citizen" + 0.011*"drug" + 0.010*"condition" + 0.008*"entry" + 0.008*"residence" + 0.008*"network" + 0.008*"request" + 0.007*"right"
2018-05-28 18:55:23,741 : INFO : topic diff=0.576249, rho=0.382192
2018-05-28 18:55:37,283 : INFO : -7.182 per-word bound, 145.2 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 18:55:37,284 : INFO : PROGRESS: pass 4, at document #3692/3692
2018-05-28 18:55:45,100 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 18:55:45,188 : INFO : topic #2 (0.031): 0.028*"animal" + 0.027*"product" + 0.019*"directive" + 0.013*"control" + 0.012*"material" + 0.011*"substance" + 0.011*"waste" + 0.009*"article" + 0.008*"import" + 0.008*"eec"
2018-05-28 18:55:45,191 : INFO : topic #30 (0.031): 0.019*"security" + 0.011*"external" + 0.010*"strategy" + 0.009*"border" + 0.008*"migration" + 0.008*"relation" + 0.008*"management" + 0.007*"defence" + 0.007*"political" + 0.007*"crisis"
2018-05-28 18:55:45,193 : INFO : topic #22 (0.031): 0.031*"fund" + 0.023*"financial" + 0.016*"education" + 0.016*"training" + 0.015*"project" + 0.011*"eur" + 0.011*"assistance" + 0.011*"million" + 0.011*"period" + 0.011*"instrument"
2018-05-28 18:55:45,194 : INFO : topic #19 (0.031): 0.066*"energy" + 0.017*"emission" + 0.016*"environmental" + 0.014*"gas" + 0.010*"electricity" + 0.010*"directive" + 0.010*"renewable" + 0.009*"source" + 0.009*"climate" + 0.009*"efficiency"
2018-05-28 18:55:45,196 : INFO : topic #29 (0.031): 0.020*"woman" + 0.018*"human" + 0.018*"child" + 0.017*"right" + 0.014*"equal" + 0.012*"health" + 0.012*"research" + 0.011*"equality" + 0.011*"discrimination" + 0.011*"man"
2018-05-28 18:55:45,197 : INFO : topic diff=0.544776, rho=0.382192
2018-05-28 18:55:45,199 : INFO : PROGRESS: pass 5, at document #2000/3692
2018-05-28 18:55:54,388 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:55:54,467 : INFO : topic #11 (0.031): 0.028*"security" + 0.019*"custom" + 0.018*"control" + 0.010*"internal" + 0.010*"export" + 0.009*"document" + 0.009*"electronic" + 0.008*"common" + 0.008*"management" + 0.007*"audit"
2018-05-28 18:55:54,470 : INFO : topic #28 (0.031): 0.026*"right" + 0.019*"datum" + 0.016*"justice" + 0.013*"judicial" + 0.013*"protection" + 0.011*"criminal" + 0.010*"freedom" + 0.009*"crime" + 0.009*"fundamental" + 0.008*"matter"
2018-05-28 18:55:54,472 : INFO : topic #12 (0.031): 0.031*"fishing" + 0.028*"fishery" + 0.018*"committee" + 0.013*"vessel" + 0.011*"ecb" + 0.010*"fish" + 0.010*"bank" + 0.009*"group" + 0.009*"central" + 0.009*"board"
2018-05-28 18:55:54,475 : INFO : topic #15 (0.031): 0.060*"directive" + 0.039*"safety" + 0.030*"health" + 0.021*"risk" + 0.015*"worker" + 0.013*"eec" + 0.012*"protection" + 0.010*"exposure" + 0.010*"requirement" + 0.010*"nuclear"
2018-05-28 18:55:54,478 : INFO : topic #4 (0.031): 0.026*"right" + 0.016*"agreement" + 0.015*"property" + 0.015*"protection" + 0.015*"online" + 0.014*"europol" + 0.013*"intellectual" + 0.013*"patent" + 0.011*"copyright" + 0.011*"enforcement"
2018-05-28 18:55:54,482 : INFO : topic diff=0.467382, rho=0.357006
2018-05-28 18:56:07,223 : INFO : -7.157 per-word bound, 142.7 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 18:56:07,224 : INFO : PROGRESS: pass 5, at document #3692/3692
2018-05-28 18:56:15,141 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 18:56:15,234 : INFO : topic #9 (0.031): 0.020*"financial" + 0.020*"crime" + 0.012*"terrorist" + 0.012*"combat" + 0.011*"prevention" + 0.010*"fraud" + 0.010*"terrorism" + 0.009*"money" + 0.009*"fight" + 0.008*"organise"
2018-05-28 18:56:15,236 : INFO : topic #19 (0.031): 0.067*"energy" + 0.018*"emission" + 0.017*"environmental" + 0.014*"gas" + 0.011*"electricity" + 0.010*"directive" + 0.010*"renewable" + 0.010*"source" + 0.009*"climate" + 0.009*"efficiency"
2018-05-28 18:56:15,238 : INFO : topic #28 (0.031): 0.026*"right" + 0.018*"datum" + 0.017*"justice" + 0.013*"judicial" + 0.013*"protection" + 0.011*"criminal" + 0.011*"freedom" + 0.009*"fundamental" + 0.009*"crime" + 0.008*"police"
2018-05-28 18:56:15,240 : INFO : topic #5 (0.031): 0.036*"regional" + 0.031*"region" + 0.030*"transport" + 0.014*"partnership" + 0.013*"network" + 0.012*"infrastructure" + 0.012*"integration" + 0.011*"partner" + 0.010*"road" + 0.010*"sea"
2018-05-28 18:56:15,242 : INFO : topic #23 (0.031): 0.044*"convention" + 0.036*"maritime" + 0.032*"ship" + 0.020*"sea" + 0.019*"port" + 0.017*"pollution" + 0.015*"marine" + 0.015*"vessel" + 0.014*"party" + 0.014*"protocol"
2018-05-28 18:56:15,244 : INFO : topic diff=0.434388, rho=0.357006
2018-05-28 18:56:15,246 : INFO : PROGRESS: pass 6, at document #2000/3692
2018-05-28 18:56:25,687 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 18:56:25,775 : INFO : topic #4 (0.031): 0.029*"right" + 0.017*"property" + 0.017*"protection" + 0.016*"agreement" + 0.015*"online" + 0.014*"intellectual" + 0.014*"europol" + 0.013*"patent" + 0.012*"copyright" + 0.011*"enforcement"
2018-05-28 18:56:25,779 : INFO : topic #11 (0.031): 0.029*"security" + 0.019*"control" + 0.018*"custom" + 0.010*"internal" + 0.010*"export" + 0.010*"electronic" + 0.010*"document" + 0.008*"common" + 0.008*"management" + 0.007*"audit"
2018-05-28 18:56:25,785 : INFO : topic #7 (0.031): 0.038*"custom" + 0.036*"tax" + 0.028*"request" + 0.020*"assistance" + 0.018*"taxation" + 0.015*"agreement" + 0.013*"legislation" + 0.011*"administrative" + 0.010*"party" + 0.010*"mutual"
2018-05-28 18:56:25,786 : INFO : topic #6 (0.031): 0.045*"product" + 0.035*"food" + 0.020*"consumer" + 0.013*"health" + 0.009*"modify" + 0.008*"agricultural" + 0.008*"agency" + 0.008*"genetically" + 0.008*"foodstuff" + 0.008*"feed"
2018-05-28 18:56:25,788 : INFO : topic #8 (0.031): 0.033*"acquis" + 0.027*"legislation" + 0.025*"progress" + 0.017*"accession" + 0.014*"field" + 0.012*"law" + 0.011*"note" + 0.011*"november" + 0.011*"effort" + 0.011*"sector"
2018-05-28 18:56:25,790 : INFO : topic diff=0.374680, rho=0.336222
2018-05-28 18:56:38,800 : INFO : -7.138 per-word bound, 140.9 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
......
2018-05-28 19:15:44,378 : INFO : topic diff=0.019104, rho=0.140240
2018-05-28 19:15:44,381 : INFO : PROGRESS: pass 49, at document #2000/3692
2018-05-28 19:15:53,096 : INFO : merging changes from 2000 documents into a model of 3692 documents
2018-05-28 19:15:53,189 : INFO : topic #18 (0.031): 0.056*"directive" + 0.042*"agency" + 0.021*"railway" + 0.021*"safety" + 0.018*"infrastructure" + 0.016*"network" + 0.016*"technical" + 0.016*"equipment" + 0.016*"requirement" + 0.015*"rail"
2018-05-28 19:15:53,192 : INFO : topic #16 (0.031): 0.018*"sector" + 0.012*"propose" + 0.012*"europe" + 0.011*"would" + 0.008*"increase" + 0.008*"could" + 0.008*"industry" + 0.008*"future" + 0.007*"change" + 0.007*"therefore"
2018-05-28 19:15:53,195 : INFO : topic #1 (0.031): 0.027*"person" + 0.014*"right" + 0.013*"citizen" + 0.013*"condition" + 0.013*"residence" + 0.012*"application" + 0.012*"noneu" + 0.012*"visa" + 0.012*"document" + 0.011*"entry"
2018-05-28 19:15:53,198 : INFO : topic #26 (0.031): 0.033*"treaty" + 0.022*"article" + 0.011*"euro" + 0.010*"procedure" + 0.010*"central" + 0.009*"rate" + 0.009*"bank" + 0.009*"government" + 0.009*"deficit" + 0.009*"budgetary"
2018-05-28 19:15:53,203 : INFO : topic #29 (0.031): 0.028*"right" + 0.025*"woman" + 0.024*"child" + 0.021*"human" + 0.018*"equal" + 0.014*"equality" + 0.014*"discrimination" + 0.013*"man" + 0.011*"gender" + 0.011*"combat"
2018-05-28 19:15:53,207 : INFO : topic diff=0.018864, rho=0.138881
2018-05-28 19:16:05,395 : INFO : -7.047 per-word bound, 132.3 perplexity estimate based on a held-out corpus of 1692 documents with 605190 words
2018-05-28 19:16:05,396 : INFO : PROGRESS: pass 49, at document #3692/3692
2018-05-28 19:16:11,532 : INFO : merging changes from 1692 documents into a model of 3692 documents
2018-05-28 19:16:11,610 : INFO : topic #20 (0.031): 0.033*"trade" + 0.032*"consumer" + 0.014*"standard" + 0.010*"business" + 0.010*"legislation" + 0.008*"practice" + 0.007*"access" + 0.007*"protection" + 0.007*"barrier" + 0.006*"internal"
2018-05-28 19:16:11,612 : INFO : topic #26 (0.031): 0.034*"treaty" + 0.022*"article" + 0.011*"euro" + 0.010*"procedure" + 0.009*"central" + 0.009*"rate" + 0.009*"government" + 0.009*"bank" + 0.009*"deficit" + 0.009*"budgetary"
2018-05-28 19:16:11,615 : INFO : topic #10 (0.031): 0.035*"financial" + 0.026*"company" + 0.024*"euro" + 0.021*"directive" + 0.018*"bank" + 0.017*"payment" + 0.016*"capital" + 0.012*"credit" + 0.012*"account" + 0.010*"institution"
2018-05-28 19:16:11,617 : INFO : topic #22 (0.031): 0.043*"fund" + 0.029*"financial" + 0.020*"project" + 0.017*"million" + 0.016*"eur" + 0.016*"assistance" + 0.015*"budget" + 0.015*"period" + 0.014*"instrument" + 0.011*"finance"
2018-05-28 19:16:11,619 : INFO : topic #11 (0.031): 0.048*"security" + 0.023*"control" + 0.021*"border" + 0.017*"defence" + 0.012*"internal" + 0.012*"common" + 0.012*"management" + 0.011*"export" + 0.011*"military" + 0.011*"external"
2018-05-28 19:16:11,621 : INFO : topic diff=0.018733, rho=0.138881

# Print topics
# I also know that none of my topic is more than 5 words i will set the num_words argument to 5
for i,topic in enumerate(ldamodel.print_topics(num_topics=32, num_words=5)):
    words = topic[1].split("+")
    print (words,"\n")

['0.045*"employment" ', ' 0.029*"labour" ', ' 0.019*"worker" ', ' 0.017*"job" ', ' 0.013*"people"'] 

['0.027*"person" ', ' 0.014*"right" ', ' 0.013*"citizen" ', ' 0.013*"application" ', ' 0.013*"residence"'] 

['0.043*"directive" ', ' 0.027*"animal" ', ' 0.015*"eec" ', ' 0.015*"product" ', ' 0.012*"substance"'] 

['0.071*"transport" ', ' 0.034*"air" ', ' 0.028*"vehicle" ', ' 0.027*"passenger" ', ' 0.022*"road"'] 

['0.066*"right" ', ' 0.034*"property" ', ' 0.026*"protection" ', ' 0.026*"intellectual" ', ' 0.017*"copyright"'] 

['0.051*"region" ', ' 0.044*"regional" ', ' 0.025*"transport" ', ' 0.019*"network" ', ' 0.016*"cohesion"'] 

['0.065*"product" ', ' 0.041*"food" ', ' 0.016*"agricultural" ', ' 0.011*"consumer" ', ' 0.010*"label"'] 

['0.061*"tax" ', ' 0.059*"custom" ', ' 0.029*"duty" ', ' 0.025*"taxation" ', ' 0.024*"request"'] 

['0.034*"acquis" ', ' 0.027*"progress" ', ' 0.026*"legislation" ', ' 0.020*"accession" ', ' 0.013*"law"'] 

['0.026*"crime" ', ' 0.021*"criminal" ', ' 0.016*"combat" ', ' 0.013*"offence" ', ' 0.012*"europol"'] 

['0.035*"financial" ', ' 0.026*"company" ', ' 0.024*"euro" ', ' 0.021*"directive" ', ' 0.018*"bank"'] 

['0.048*"security" ', ' 0.023*"control" ', ' 0.021*"border" ', ' 0.017*"defence" ', ' 0.012*"internal"'] 

['0.035*"fishery" ', ' 0.033*"committee" ', ' 0.029*"fishing" ', ' 0.016*"vessel" ', ' 0.016*"statistic"'] 

['0.038*"directive" ', ' 0.027*"law" ', ' 0.020*"court" ', ' 0.013*"contract" ', ' 0.013*"legal"'] 

['0.060*"education" ', ' 0.045*"training" ', ' 0.025*"people" ', ' 0.021*"young" ', ' 0.020*"youth"'] 

['0.063*"health" ', ' 0.047*"safety" ', ' 0.033*"nuclear" ', ' 0.030*"risk" ', ' 0.024*"directive"'] 

['0.018*"sector" ', ' 0.012*"propose" ', ' 0.012*"europe" ', ' 0.011*"would" ', ' 0.009*"increase"'] 

['0.027*"research" ', ' 0.024*"strategy" ', ' 0.015*"innovation" ', ' 0.014*"plan" ', ' 0.013*"technology"'] 

['0.056*"directive" ', ' 0.042*"agency" ', ' 0.021*"safety" ', ' 0.021*"railway" ', ' 0.019*"infrastructure"'] 

['0.082*"energy" ', ' 0.025*"emission" ', ' 0.022*"environmental" ', ' 0.017*"gas" ', ' 0.016*"water"'] 

['0.033*"trade" ', ' 0.032*"consumer" ', ' 0.014*"standard" ', ' 0.010*"business" ', ' 0.010*"legislation"'] 

['0.126*"agreement" ', ' 0.022*"party" ', ' 0.021*"research" ', ' 0.011*"joint" ', ' 0.011*"part"'] 

['0.043*"fund" ', ' 0.029*"financial" ', ' 0.020*"project" ', ' 0.017*"million" ', ' 0.016*"eur"'] 

['0.063*"convention" ', ' 0.038*"maritime" ', ' 0.033*"ship" ', ' 0.022*"sea" ', ' 0.019*"port"'] 

['0.027*"paper" ', ' 0.023*"medium" ', ' 0.022*"audiovisual" ', ' 0.022*"green" ', ' 0.020*"general"'] 

['0.080*"aid" ', ' 0.028*"competition" ', ' 0.019*"article" ', ' 0.013*"grant" ', ' 0.012*"guideline"'] 

['0.034*"treaty" ', ' 0.022*"article" ', ' 0.011*"euro" ', ' 0.010*"procedure" ', ' 0.009*"central"'] 

['0.062*"datum" ', ' 0.026*"access" ', ' 0.021*"electronic" ', ' 0.017*"network" ', ' 0.015*"internet"'] 

['0.025*"right" ', ' 0.023*"justice" ', ' 0.016*"asylum" ', ' 0.015*"freedom" ', ' 0.015*"judicial"'] 

['0.028*"right" ', ' 0.025*"child" ', ' 0.025*"woman" ', ' 0.021*"human" ', ' 0.019*"equal"'] 

['0.014*"strategy" ', ' 0.012*"dialogue" ', ' 0.011*"security" ', ' 0.011*"partnership" ', ' 0.010*"political"'] 

['0.033*"progress" ', ' 0.027*"accession" ', ' 0.024*"candidate" ', ' 0.023*"acquis" ', ' 0.016*"turkey"']

# Now lets save the model, dictonary, and corpus to use it for further use
# Saving Model
ldamodel.save('/home/jay/ANN_Models/LDA/LDA_Model')

# Saving Corpus
import pickle
with open('/home/jay/ANN_Models/LDA/LAD_Corpus.pickle', 'wb') as p:
    pickle.dump(cleaned,p)
    
# Saving Dictonary
dictionary.save('/home/jay/ANN_Models/LDA/LDA_Dictonary')

2018-05-28 19:17:04,097 : INFO : saving LdaState object under /home/jay/ANN_Models/LDA/LDA_Model.state, separately None
2018-05-28 19:17:04,111 : INFO : saved /home/jay/ANN_Models/LDA/LDA_Model.state
2018-05-28 19:17:04,120 : INFO : saving LdaModel object under /home/jay/ANN_Models/LDA/LDA_Model, separately ['expElogbeta', 'sstats']
2018-05-28 19:17:04,122 : INFO : storing np array 'expElogbeta' to /home/jay/ANN_Models/LDA/LDA_Model.expElogbeta.npy
2018-05-28 19:17:04,126 : INFO : not storing attribute id2word
2018-05-28 19:17:04,127 : INFO : not storing attribute state
2018-05-28 19:17:04,128 : INFO : not storing attribute dispatcher
2018-05-28 19:17:04,132 : INFO : saved /home/jay/ANN_Models/LDA/LDA_Model
2018-05-28 19:17:04,848 : INFO : saving Dictionary object under /home/jay/ANN_Models/LDA/LDA_Dictonary, separately None
2018-05-28 19:17:04,856 : INFO : saved /home/jay/ANN_Models/LDA/LDA_Dictonary

Notebooks

You can view this notebook here

LSTM with 8 Classes

Fri, 25 May 2018 00:00:00 +0000

New LSTM with 8 classes

In this script I will be using only 8 out of 32 classes that were originaly present in the dataset, this is necessary because the data in other classes is much less compared to these 8 classes, this makes it difficult for the neural network to learn anything off of those classes.

import pickle
from gensim.models import Word2Vec
import keras
import numpy as np
import pandas as pd
from keras.datasets import imdb
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.layers import LSTM
from keras.layers.embeddings import Embedding

/home/jay/.local/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.

I have already stored the senteces and their corresponding labels as stated in this post. Hence I am going to use those sentences and labels.

# Loading data
with open('/home/jay/pickled/sent.pkl', 'rb') as f:
    sents = pickle.load(f)
    
with open('/home/jay/pickled/label.pkl', 'rb') as g:
    label = pickle.load(g)

As the model was not learning, I have decided to reduce the number of classes to 8, so I will pick up top 8 classes with maximum number of samples.

# lets count number of sample in each class
from collections import Counter

counter = Counter()

for l in label:
    counter[l] +=1

As I have counted number of samples in each class, I will pick out top 8 classes from them and use only the 8

import heapq
classes = heapq.nlargest(8,zip(counter.values(), counter.keys()))
classes

[(19018, 4),
 (15679, 24),
 (12455, 15),
 (10219, 3),
 (10011, 0),
 (7008, 26),
 (6606, 5),
 (6392, 11)]

These are the classes I am going to use for training the neural network.

sample_8 =[]
label_8 = []

for label, sent in zip(label, sents):
    if (label == 4) or (label ==24) or (label == 15) or (label==3) or (label==0) or (label==26) or (label==5) or (label==11):
        sample_8.append(sent)
        label_8.append(label)

len(sample_8)

As I have stated in my Data Descrepancies post that some sentences are very long. This is a problem when traning recurrent neural network with long sequences. For example think of a situation where you have a sentence of 10 words and another sentence which is 200 word sentence which also happens to be the maximum length of sentence in dataset. So when you are padding all the sentences the sentence with 10 words will have 190 padded zeros, so you see how this is very inefficient and that there has to be some solution to things like this. To tackle this problem what I think is a easy and effective solution is to break the 200 word sentence into number of smaller sentneces. This is however not ideal as we are loosing semantics and realations between words but this seams to be the only viable option right now. There are other options which you can learn about it here.

So for me what I will do is I will use sliding window or rolling window algorithm. To preserve the context of words in the longer sequence I will take smaller steps. For example, I have a sentence of 100 words, so I will first take a window of 20 words and make it into a sentence, then I will move five words from the starting and take another 20 words. So in my first sentence I will have words from index 0-19 and in my second sentence i will have words from index 4-24 and so on. This helps in preserving the context of the sentences.

The setting I used is winSize=20 and step=10

def slidingWindow(sequence,winSize,step):
    """Returns a generator that will iterate through
       the defined chunks of input sequence. Input sequence
       must be sliceable."""
    
    # Pre-compute number of chunks to emit
    numOfChunks = round(((len(sequence)-winSize)/step)+1)

    # Do the work
    for i in range(0,numOfChunks*step,step):
        yield sequence[i:i+winSize]

Make new sentece and labels list and check the length of every sentnce in sample_8, if there are more than 20 words in the sentence apply sliding window on it and store them into new list

new_sent = []
new_label = []

for s, l in zip(sample_8, label_8):
    if (len(s.split()))<=20:
        new_sent.append(s)
        new_label.append(l)
    if (len(s.split()))>20:
        slides = slidingWindow(s, 20, 10)
        for slide in slides:
            new_sent.append(' '.join(slide))
            new_label.append(l)

len(new_sent)

# Defining max_length
max_sent = max(new_sent, key=len)    # Get the longest sentence in the list
max_length = len(max_sent.split())  # split it and set the max_length
max_length

# Using Keras tokenizer
# Here I am using 10000 words to keep based on the word frequency

tokenizer = keras.preprocessing.text.Tokenizer(num_words=10000)

# Tokenizing the sentences (This process may take some time depending on your corpus size)
tokenizer.fit_on_texts(new_sent)

# Lets see what our vocabulary size is
vocab_size = len(tokenizer.word_index) + 1   # We are adding 1 here because it takes indexing from zero
vocab_size

Even though it shows the vocab size to be 15532 it will only use 10000 words as we have spacified above.

# Sentence encoding
sent_encoded = tokenizer.texts_to_sequences(new_sent)

# Start padding with the max_length
padded_sents = keras.preprocessing.sequence.pad_sequences(sent_encoded, maxlen=max_length, padding='post')
# Note: We are using post padding

# Split the data into test and train
from sklearn.model_selection import train_test_split

train_set, test_set, train_label, test_label = train_test_split(padded_sents, label_8,test_size=0.33)

from sklearn.preprocessing import OneHotEncoder

# Define one_hot_encoder object
onehot_encoder = OneHotEncoder(sparse=False)

train_labels = onehot_encoder.fit_transform(np.reshape(train_label,(-1,1)))
test_labels = onehot_encoder.fit_transform(np.reshape(test_label,(-1,1)))

As I know that classes are unbalanced, the most common parctice is to use class weights to award penalty to a wrong prediction. For example, if we have 2 classes with class_1 = 80 and class_2 = 20 samples, then we can award a penalty to our network if the sample being predicted is of class_2 and is predicted to class_1. To calculate penalty we take a reference class, I usually take the class with maximum samples, and then calculate penalty for other classes with this reference class. The penalty for class_1 = (no. of sample in reference class)/(no. of sample in class_1) which for our example will be 80/80=1 and for class_2 = 80/20 = 4.0

# Computing the class weights
class_ = []
weights = []

ref_class = classes[0][0]

for i, j in enumerate(classes):
    class_.append(i)
    weights.append(ref_class/j[0])

# Creating dictonary out of the two lists
class_weight = dict(zip(class_, weights))

class_weight

{0: 1.0,
1.2129600102047324,
1.5269369731031714,
1.8610431549075253,
1.8997103186494855,
2.713755707762557,
2.8788979715410234,
2.9752816020025032}

# Create sequential model
model = Sequential()
reduce_rate = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=0, verbose=1, 
                                                mode='auto', epsilon=0.0001, cooldown=0, min_lr=0)

early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, 
                                           patience=1, verbose=1, mode='auto')

model.add(Embedding(vocab_size, 1000, input_length=max_length))   # adding embedding layer, which we have defined earlier

model.add(LSTM(30,recurrent_regularizer=keras.regularizers.l2(0.1),return_sequences=True))     # LSTM layer 
model.add(Dropout(0.1))
model.add(LSTM(50,recurrent_regularizer=keras.regularizers.l2(0.06),return_sequences=False))
#model.add(Dropout(0.5))
model.add(Dense(8, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer= 'adam', metrics=['accuracy'])
print(model.summary())
model.fit(train_set, train_labels, validation_split=.1, epochs=10, batch_size=512, 
          verbose=1,class_weight=class_weight ,callbacks=[reduce_rate,early_stop])

/home/jay/.local/lib/python3.5/site-packages/keras/callbacks.py:928: UserWarning: `epsilon` argument is deprecated and will be removed, use `min_delta` insted.
  warnings.warn('`epsilon` argument is deprecated and '
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 20, 1000)          15356000  
_________________________________________________________________
lstm_1 (LSTM)                (None, 20, 30)            123720    
_________________________________________________________________
dropout_1 (Dropout)          (None, 20, 30)            0         
_________________________________________________________________
lstm_2 (LSTM)                (None, 50)                16200     
_________________________________________________________________
dense_1 (Dense)              (None, 8)                 408       
=================================================================
Total params: 15,496,328
Trainable params: 15,496,328
Non-trainable params: 0
Train on 68274 samples, validate on 7587 samples
Epoch 1/10
68274/68274 [==============================] - 16s 227us/step - loss: 5.3191 - acc: 0.3646 - val_loss: 2.8831 - val_acc: 0.5347
Epoch 2/10
68274/68274 [==============================] - 13s 192us/step - loss: 2.3372 - acc: 0.5942 - val_loss: 2.1347 - val_acc: 0.6038
Epoch 3/10
68274/68274 [==============================] - 13s 193us/step - loss: 1.8782 - acc: 0.6433 - val_loss: 2.0397 - val_acc: 0.6060
Epoch 4/10
68274/68274 [==============================] - 13s 194us/step - loss: 1.6965 - acc: 0.6656 - val_loss: 2.0175 - val_acc: 0.6137
Epoch 5/10
68274/68274 [==============================] - 13s 193us/step - loss: 1.5890 - acc: 0.6803 - val_loss: 2.0452 - val_acc: 0.6171

Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.
Epoch 00005: early stopping

scores = model.evaluate(test_set, test_labels, verbose=1)

print("Accuracy: %.2f%%" % (scores[1]*100))

37365/37365 [==============================] - 23s 622us/step
Accuracy: 61.45%

# Classification Report (Precision, Recall and F1-Score)
from sklearn.metrics import classification_report, confusion_matrix
classificationReport = classification_report(y_true, y_pred)

print(classificationReport)

print(confusion_matrix(y_true, y_pred))

precision    recall  f1-score   support

          0       0.73      0.47      0.57      4268
          1       0.67      0.45      0.54      4169
          2       0.79      0.67      0.73      8563
          3       0.60      0.52      0.56      2816
          4       0.52      0.46      0.49      2400
          5       0.62      0.67      0.64      5409
          6       0.46      0.75      0.57      6799
          7       0.66      0.68      0.67      2941

avg / total       0.64      0.61      0.62     37365

[[1989  106  147   80  245  769  890   42]
 [  90 1886  370  130  287  230 1039  137]
 [  58  215 5753  380  127  297 1506  227]
 [  60   95  376 1469   52  155  537   72]
 [ 114  185  101   39 1108  316  506   31]
 [ 309   49  142  102  206 3635  882   84]
 [  87  239  272  168   99  399 5114  421]
 [   3   21   99   66   22   78  645 2007]]

Curating the Dataset again!

Wed, 23 May 2018 00:00:00 +0000

Curating Dataset

After the initial results it was clear that I have a lot less data then what the neural networks need to perform better. So I have decided that I will use everything in the files as my training data instead of removing everything except SUMMARY of the document. Also I will remove all the line with all the captial letters in order to remove the tags.

import os
import sys
import re
import string
from string import punctuation
from nltk.corpus import stopwords
import spacy

# NLP model from spacy
nlp = spacy.load('en')

# First lets create list of topics
label = []
for root, dirs, files in os.walk('/home/jay/GITHUB/Data_Thesis_1/Ready/'):
    label.append(dirs)

labels = dict(list(enumerate(label[0])))

def preprocess(line):
    """Preprocess given line.
    :params: line as string
    :return: manipulated line as string
    """
    
    # lower case
    lower = line.lower()
    
    t = re.sub(r"(\d+\.\d+)","",lower)
    t = re.sub(r"\d{2}.\d{2}.\d{4}","",t)
    t = re.sub(r"\d{2}\/\d{2}\/\d{4}","",t)
    t = re.sub(r"\d{2}(\/|\.)\d{2}(\/|\.)\d{2}","",t)
    t = re.sub(r"($|€|¥|₹|£)","",t)
    t = re.sub(r"(%)","",t)
    t = re.sub(r"\d+","",t)
    t = re.sub(r"\n","",t)
    t = re.sub(r"\xa0", "", t)
    
    # Removing punctuations
    
    table = str.maketrans("","", string.punctuation)
    text = t.translate(table)
    
    
    # Removing other extras
    t = re.sub(r"\"|\—|\'|\’|–","",text)
    
    # Removing stop words
    
    stop_words = stopwords.words('english')
    t_ = [word for word in t.split() if word not in stop_words]
    t = ' '.join(t_)
    
    
    # Lemmetizer
    sent = []
    doc = nlp(t)
    for word in doc:
        sent.append(word.lemma_)
        
    text = " ".join(sent)
    
    return text

caps_with_question_mark = re.compile(r"^([A-Z ':]+\?$)", re.M)
caps_without_question_mark = re.compile(r"^([A-Z ':]+$)", re.M)

# Iterate over files and get the content.
sents = []
label = []
for root, dirs, files in os.walk('/home/jay/Thesis_1/Data/Data_EN/'):
    for file in files:
        if file.endswith('.txt'):
            topic = root.split(os.path.sep)[-2] 
            if file != 'log.txt':
                with open(os.path.join(root, file)) as f:
                    content = f.read()
                    
                    
                    for key,value in labels.items():
                        if str(value) == str(topic):      # If the dir is equal to any value in the labels dict then
                            label_ = key
                    
                    upp = caps_with_question_mark.sub('',content)
                
                    upp_ = caps_without_question_mark.sub('',upp)
                
                    # remove first two lines   
                    cont = upp_.split('\n')
                    cont = cont[2:]
                    
                    # preprocess every line
                    lines = []
                    for line in cont:
                        lines.append(preprocess(line))
                    line_ = []    
                    for l in lines:
                        line_.append((re.sub('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)', '',l)))
                    

                    
                    _line = []
                    for l in line_:
                        k = l.replace('oj', '')
                        n = k.replace('ii', '')
                        _line.append(n)
                        
                    final = [line for line in _line if line]
                    
                    
                    for line in final:
                        sents.append(line)
                        label.append(label_)
                    
                  
                    
                    

len(sents), len(label)

(164679, 164679)

import pickle
with open('/home/jay/pickled/sent.pkl', 'wb') as f:
    pickle.dump(sents,f)
    
with open('/home/jay/pickled/label.pkl', 'wb') as g:
    pickle.dump(label,g)

	5	...
0	0	...
1	4	...
2	1	...
3	9	...
4	2	...

	5	...
0	0	...
1	4	...
2	1	...
3	9	...
4	2	...

	5	...
0	0	...
1	4	...
2	1	...
3	9	...
4	2	...