LISDATA Develops a Topic Modeling Tool to Analyze Forums

Knowing what topics are discussed on a social network or understanding the direction of user conversations in an internet forum on a specific issue is not easy: the information is scattered, spread across thousands of comments, and it is very difficult to perform a manual analysis of it.

A Solution Based on Machine Learning

To address the challenge of analyzing information contained in web forums, LISDATA applied our knowledge in big data and machine learning. The project, for which an estimated duration of 12 months was set, presented a practical approach to some of our previous research on Natural Language Processing (NLP) techniques.

The goal was to perform an automatic analysis of a set of Spanish texts (in this case, comments in opinion forums) to identify discussion topics and obtain a summary of the most prominent ones. It also needed to be possible to search for the presence of specific topics, perform analyses at different levels, or suggest extensions to topics proposed by users.

How to Use Topic Modeling to Generate Understandable Information

Our R&D team decided to use machine learning techniques such as “topic modeling,” which involves detecting patterns in word usage and consequently grouping text pieces based on those patterns.

We created a corpus with Amazon comments to train the model on forum writing style (informal, with spelling mistakes, short and simple sentences, etc.) and tested different topic modeling and lexical embedding techniques. By the end of the project, we had achieved our goal: we had a system that performed a satisfactory analysis of the main topics of each conversation and presented the data in an organized and easily understandable way on a dashboard.

Financial Data