It was noted that, at the time of evaluation, query is not scored against all the documents available, and it was squarely evaluated on the only two negative examples for every one positive examples. When a query is scored against all available documents, the accuracy dropped to slightly better than random. In a different experiment, ELMO embeddings was used. An ELMO model has to be tried remove one unveriffiable factor (embeddings used in this model). Since memory issues were observed while using ELMO embeddings, generators have to be used to load one batch at a time at the time of training

• Initial SiameseLSTM model experiment done by using the following data set to understand the entire architecture.

https://github.com/amansrivastava17/lstm-siamese-text-similarity/blob/master/sample_data.csv

Notes :

Inputs for the LSTM model are Query text, Document text, Feedback / User’s response. User’s response can be ‘0’ or ‘1’ depending on whether the user accept / reject the recommendation.

Output will be a similarity score between Query and document.

Parameters used for the LSTM model are

EMBEDDING_DIM = 50

MAX_SEQUENCE_LENGTH = 10

VALIDATION_SPLIT = 0.1

RATE_DROP_LSTM = 0.17

NUMBER_LSTM = 50

NUMBER_DENSE_UNITS = 50

ACTIVATION_FUNCTION = ‘relu’

• Based on the Query - Document features, User response data on the Content Recommendation and the above parameters, LSTM model performance was very poor. Reasons can be:

1. There was an imbalance between the number of negative and positive responses. Basically data set was not a balanced one.

2. Very few common words between Query - Doc pairs.

3. Loss function used in the base model was Binary Cross Entropy. In the case of imbalance situation we could have chosen Categorical Hinge Fun.

4. Word2Vec representation of text will not be a good vector representation for the data set.

• Based on these hypotheses, revised the model and important findings / thoughts are

1. Instead of Binary cross entropy, checked with Categorical Hinge function. Model prediction score seems to be almost similar for both loss function. So decided to go with Binary cross entropy which is already in the model.

2. Input for the LSTM model was the vector representation of Document - Query keywords. Instead of considering only keywords, to increase the number of common words between query - doc pair, add more features like titles, subtitles, description to each document.

3. To reduce imbalance in the data set, restructured the data set where each query have 1 positive response and 2 negative responses.
   A sample data set and LSTM results are sharing here. https://docs.google.com/spreadsheets/d/1yOnYG5hmJpi9LFSA20Z6YDNb3tUbH8ivwXcZqJTp4nE/edit#gid=0

Findings: After a revision in the model based on the above hypotheses, prediction score has improved.

True Positive Rate = 99.82 %

True Negative Rate = 99.77 %

Accuracy = 99.79 %

( True positives = 1129, True negatives = 2248, False positives = 5, False negatives = 2 )

Thoughts: Balanced distribution of positive negative response in the data set can be the main reason for the better performance of the model. Since high accuracy leads to an ambiguous situation, need to revisit the entire Siamese LSTM procedure.