rubylexer 0.7.7

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant as a lexer for Ruby that's complete and correct; all legal Ruby code should be lexed correctly by RubyLexer as well. Just enough parsing capability is included to give RubyLexer enough context to tokenize correctly in all cases. (This turned out to be more parsing than I had thought or wanted to take on at first.) RubyLexer handles the hard things like complicated strings, the ambiguous nature of some punctuation characters and keywords in ruby, and distinguishing methods and local variables. RubyLexer is not particularly clean code. As I progressed in writing this, I've learned a little about how these things are supposed to be done; the lexer is not supposed to have any state of it's own, instead it gets whatever it needs to know from the parser. As a stand-alone lexer, Rubylexer maintains quite a lot of state. Every instance variable in the RubyLexer class is some sort of lexer state. Most of the complication and ugly code in RubyLexer is in maintaining or using this state.