Back then I used a nice video from some nice guy but somehow could not find the video with search engines anymore. Too bad. Implemented LDA in Java back then based on that tutorial. I learned how it works, not why it works. Actually I found a reasonably good explanation on Quora.
Well, it is a good one if you already know most of how LDA works. Also a tutorial briefly summarizing how online LDA works, which is a nice improvement, and I guess what the tools use these days. The number of topics LDA produces is given as a parameter, and is always a bit of a puzzle for me how to pick the best number for topics.
Maybe some of the libs out there will do it for me? Python seems all the rage in data science these days, because whatever. So after a few search, gensim it is. Gensim seems to have some perplexity options and a bunch of weird formulas to apply. Is it so hard to write some simple docs and explain these things? I guess nobody pays people to do it, and doing for free would just go against the goal of making oneself important.
Sort of makes sense, and applies to most OSS software I have used. Or maybe I am just bad at using stuff. There is also something called topic coherence in Gensim. This is supposed to be some way to evaluate the number of topics. Somehow the explanation does not work for me. I did not quite grasp how it works for real. So I just gave it a try to see what I get, that would be most important for me regardless.
I start with the English wikipedia I used a May dump. Gensim nicely comes with a script to parse it for dictionary and corpus:. Then some code to build different sizes of topic models 25 to topics in 25 topic size increments.
