I have a code that reads tokens from a text. I'm using Visual Studio 2013 in an extensibility project.

public class TestScanner : IScanner
{
    private TestSource m_source;
    private int m_offset;

    public TestScanner(LanguageService service, IVsTextLines buffer)
    {
        this.m_source = new TestSource(service, buffer, new TestColorizer(service, buffer, this));
    }

    public bool ScanTokenAndProvideInfoAboutIt(TokenInfo tokenInfo, ref int state)
    {
        bool bFound = false;
        if (tokenInfo != null)
        {
            bFound = this.m_source.GetNextToken(m_offset, tokenInfo, ref state);
            if (bFound)
            {
                m_offset = tokenInfo.EndIndex + 1;
            }
        }
        return bFound;
    }

    public void SetSource(string source, int offset)
    {
        m_source.SetText(source);
        m_offset = offset;
    }
}

I used this link to verify the utility of the interface:

  

IScanner.ScanTokenAndProvideInfoAboutIt Method

     

Returns true if the token was parsed from the current line and   information returned; otherwise, returns false indicating no more   tokens on the current line.

I'm not using the class directly in the code, I implement the interface through the TestScanner class and VS uses it to read the code (although I have Colorizer , but with no override ).

What I hope to happen:

• SetSource send me the lines of code of the text, I recognize the tokens and, when there is no more, jump to next line, as it says in the documentation, just return false in ScanTokenAndProvideInfoAboutIt .

What happens

• In fact, ScanTokenAndProvideInfoAboutIt faults returns true the first time, and the value of m_offset is modified, when passing the second method, m_offset has the value of the end of the line (13 specifically, using the string "% with% ")
• As using System; is equal to or greater than the string this.m_source.GetNextToken returns false, sequentially in m_offset , that is, indicate that the collection and acknowledgment of the tokens has been terminated.
• But ScanTokenAndProvideInfoAboutIt sends me back the same string as the last collection of tokens, thus entering an infinite loop.

I would like to know why SetSource repeats the call with the same argument (same row) even though I returned false. Is it some Visual Studio bug?

What logic do I need to use to say that I have just recognized the line Tokens?

Note: If you need more code, just comment!