Skip to main content
Software Engineering

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

11
  • 2
    The only problem I foresee is that the probabilities will be vastly smaller than in your toy example. Given numerical instability, this means that soon all probabilities are 0. Using log odds solves this though. Furthermore, I’d use larger tokens (i.e. not characters but words/punctuation). Commented Jun 28, 2011 at 15:20
  • 2
    @Konrad: the idea here isn't to test absolute probabilities: it's to test relative probabilities. For each line, is the text of that line more likely to have been generated by an English language model, or by a code language model. Commented Jun 28, 2011 at 15:48
  • 5
    You can train this model on existing SO posts (particularly because you may need to account for Markdown syntax). If you assume that most posts are formatted correctly (or you pick through a large number of posts, on the order of tens of thousands, to remove posts that are not formatted correctly), then you assume that stuff that's not code formatted is English text, and stuff that is code formatted is code, you can train from actual SO answers. Commented Jun 28, 2011 at 15:51
  • 1
    A tutorial about how to do this (using LingPipe in Java) is available from LingPipe's website. At the end of the tutorial, there are a number of papers on techniques to tackle this problem. I suggest reading them. Commented Jun 28, 2011 at 16:04
  • 1
    It’s interesting to see that the state of the art solution has only a very low vote count, and rates vastly less than all those ad-hoc solutions which, admittedly, might just be good enough but rely a lot on special-casing and are inherently prone to overfitting. Commented Jul 3, 2011 at 10:56