Incorrect Scanner Generated with BABEL option set.

May 12, 2010 at 9:57 AM

I've just kicked back into programming again after a few years off and am toying with your products (GPPG/GPLEX), perhaps with the option of making some new languages. They're great fun to play with and pretty clearly coded - so easy to follow. The first thing I wanted to do was take your two demo projects and adapt them with Babel to create a syntax highlighting add-in for Visual Studio 10. The good news is I managed to put the two projects together and the .lex and .y files are now glowing green, blue and red. Unfortunately, it was much harder than it really should have been.

Most things I can hack around, but one needs a change to the toolchain (which I can do, but then I've using a branched toolset).

The worst issue is the namespace collisions that occur when Babel code and GPLEX code meet. They share many of the same class names and definitions, which for the most don't clash. There are some clashes though - the worst being in Scanner.cs - the generated lexer code. When you enable Babel and set the define in your project you get a line similar to

internal sealed partial class Scanner : ScanBase, IColorScan

The problem is that IColorScan resolves to a copy within the QUT code, and when this routine is called

public LineScanner ()
	this.lex = new QUT.GPGen.Lexers.Scanner ();

it expects Scanner to be implementing Babel.ParserGenerator.IColorScan. This prevents me from allowing the build system to re-create the Scanner.cs each time because I have to apply this tweak every time.

A second collision seems to occur on the ShiftReduceParser, one of which is supplied by Babel. I managed to find it and switch it off, but it would be better if there was no conflict.

Finally, I had to make numerous little fix-ups in Babel and the invariant part of GPLEX/GPPG in order to make them play nice. Most involved dis-disambiguating which class an object was - or moving from a Babel specific version of a class to the more applicable GPLEX/GPPG version. I'm aware both projects are moving targets right now, so am loathe to make a swathe of changes which will just need to be applied again then patches come out. Still, I'd be happy to contribute patches to your project.

The good news is, I managed to get both extensions written, and quickly too (2 days), and I'm happy to pass on either the code or the extensions for anyone wanting syntax highlighting while they hack their .lex and .y files.

May 13, 2010 at 1:06 PM

Hi blackhawk451.  Thanks for the feedback.  I have not tested the last few iterations of the tools against the Babel code, which I understand has changed since I last worked on it.

The best way to progress the issues that you raise is to send me the example code that is causing the problems.  There are a number of mechanisms in the tools to change the default naming of features, so I need to know if these mechanisms are inadequate, of it some defaults need to change with the /babel option. 

If we transfer the discussion to the issues tab you can upload files demonstrating the issues.  Alternatively I can pass on my email address to receive a zip of the code. 



May 15, 2010 at 3:01 PM

Thanks for the reply John. I'm currently re-factoring my changes again to cause the least change to both sets of code, and it's been quite the trip. I've tried moving the classes into the Babel namespace, but that required large amounts of edits - and I've since reversed most out. Keeping them in the QUT.Gplex.* (I prototyped off the GPLEX code) namespaces reduces the number of changes required to GPLEX but causes situations I can't resolve without changes in Babel (which I am trying to minimise).

My current position is that I have just two files I need to change in Babel (LanguageService, LineScanner), and four within GPLEX (AAST, ErrorHandler, ParseHelper, ScanHelper). Most of the changes on the side of GPLEX are what you'd expect to do, since they are tied to the language specification. I changed ErrorHandler to get it's strings from resource files instead because the huge case statements made my head hurt :-)

I'll have a working build again in a day or so and will post it when it's ready. My goal has been to reduce the amount of dependancies and reduce the surface area friction of the projects. The end goal of course being to make it easy to whip up a new project - with a wizard, and get it ready to compile / syntax highlight straight away.

A side goal is to get the two toolchains of GPLEX and GPPG upgraded to provide syntax highlighting and completion for when I'm editing. I have this working now - but GPPG needs improvement, it's a bit flaky on the /* comments */ and some other parts.

It's been great fun playing with this codebase. I haven't thought about this stuff since I was in uni, which oddly enough is the one you're at :D <googles> Actually, I think you might be an old lecturer of mine. I was a student in 1987 and learnt Pascal and Modula 2 there (used that for years, until switching to C++).


Ivan Hawkes

May 15, 2010 at 3:31 PM

Oh, to answer the initial post, it's an error of omission in the declaration of the Babel specific version of  IColorScan. This interface declaration should inhereit from Babel.IColorScan and that will fix everything. Polymorphism is great.

Change line 233


Console.WriteLine ("public interface IColorScan  : IColorScan {");


Console.WriteLine ("public interface IColorScan  : Babel.ParserGenerator.IColorScan {");
You can snip the following lines for SetSource and GetNext too, they won't need declaring twice though it produces no errors.