Scilexer.dll for OxygenBasic?

[Arnold]

Hi Charles,

maybe I can persuade you to use a more current version of Scilexer.dll? After exploring this topic a little bit I found that there are many more options now.
I found a link to an excellent tutorial about adding a lexer to Scintilla:
https://groups.google.com/forum/#!topic/scintilla-interest/zzBltE93d8o

This was very helpful. I downloaded the source of Scintilla version 3.74 and modified /include/Scintilla.iface and /lexers/LexBasic.cxx. I then deleted all files in /lexers except LexNull.cxx and LexBasic.cxx. After running python with /scripts/Lexgen.py, HFacer.py I got a modified /include/SciLexer.h and /src/Catalogqu.cxx. I used msys with gcc/g++ version 4.8.1 to run the makefile in /win32 and I now have a scilexer.dll which is only suitable with oxygenbasic/purebasic/blitzbasic/powerbasic/freebasic.

I used the two files with Scintilla version 3.61 too. This will give a more compact dll. The size for scilexer.dll 3.61 is 605 kb, the size for scilexer.dll 3.74 is 1021 kb. I suppose with Visual Studio 2015 the size of the exe files would be more compact but I have no access to this compiler. Maybe there is someone in this list who could test this with Visual Studio? I will attach the necessary files.

There are two issues. I did not add ';' as a comment because Oxygenbasic can treat this as a comment but also can treat this C-like.
There is no method in LexBasic.cxx to treat Rem as a comment. But as the other basic dialects use rem too I suppose there is a special way to hightlight Rem as a comment statement.

There is also another idea with LexBasic.cxx. As I am really no C-programmer I cannot do this. But if you would check the code for correctness and possible modifications you could contribute it to the Scintilla / Scite project. SciTE (which has been enhanced too in the meantime) could then be used natively with Oxygenbasic.

I have attached the two scilexer.dlls and the necessary files for recreating the dlls.

Roland

Edit: Deleted and attached updated version 3.74 with later post

[Arnold]

And I know that it works.

These are the lines which I modified in Oxide.o2bas:

Code: [Select]

...
    'USE SCINTILLA
    if sci then
      hchw(0) = CreateWindowEx(0,"Scintilla",null,EditStyle,0,0,0,0,hwnd,id+i,inst,null)
      hsci=hchw[0]
      % SCLEX_OBASIC        40
      '
      def sm SendMessage hsci,
      '
     'http://search.cpan.org/dist/Wx-Scintilla/lib/Wx/Scintilla/Constant.pm#Lexical_states_for_SCLEX_CPP
     'http://www.garybeene.com/code/gbsnippets_gbs_00663.htm
     '3  cpp
     '8  vb ***
     '40 lout
     '51 powerbasic
     '75 freebasic
//      sm SCI_SETLEXER, 8, 0
      sm SCI_SETLEXER, 200,0    'Oxygenbasic lexer
      include "oxykeywords.inc"
      sm SCI_SETKEYWORDS, 0, strptr OxyKeywords
//      sm SCI_COLOURISE, 0, -1
      '
     'sm SCI_STYLESETFORE, SCE_OB_DEFAULT,0
     'sm SCI_STYLESETBACK,32,0xFFFFFF  'white
     'sm SCI_STYLECLEARALL, 0, 1
//      for i=0 to 8
      for i=0 to 31
        sm SCI_STYLESETFONT, i, fnt 'Courier / Lucida Console
        sm SCI_STYLESETSIZE, i, fnz '10..14 point size range
      next
      '
      sm SCI_SETMARGINTYPEN,  0, SC_MARGIN_NUMBER
      sm SCI_SETMARGINWIDTHN, 0, 30
      sm SCI_SETCARETLINEBACK,0xffeeee,0
      sm SCI_SETCARETFORE,    0xff4444,0
      sm SCI_SETCARETLINEVISIBLE,1,0
      '
      def sn SendMessage hsci, SCI_STYLESETFORE,
      '
      'setting RGB colors
      sn 0,  0xff0000  '
      sn 1,  0x207800  'line comment
      sn 2,  0x006000  'numbers
      sn 3,  0xd00000  'keywords
      sn 4,  0x008800  'string
      sn 5,  0xc00080  '#names
      sn 6,  0x000088  'symbols
      sn 7,  0x002000  '

      sn 23,  0x207800  'block comment
      sn 24,  0x207800  'commentline //
      '
      SetText hsci,s
      '
    else 'NOT SCINTILLA
...

The lines after // and the statements for block comment and commentline was all I did. For some reason I had to comment out #compact in Oxide.o2bas to run BuildInstallOxide.bat. This will already give basic highlighting. There is much more possible with Scintilla. Of course sintilla.inc must be adapted a little bit (and most of the SCI_SETLEX constants would not be necessary) to use the more advanced capabilities, but it already works fine with OxIde.

.

[Charles Pegge]

Hi Roland,

Many thanks for all your work on O2HEdit and Scintilla.

Let's adopt your binary of version 3.61, and modifications for Oxide, and I'll look into the problem with #compact

This is some way off in the future but my ambition is to use Oxygen's own lexing and parsing system to feed information back to the editor, and the Help system by partial compiling. This would, as far as I know, fall outside the scope of Scintilla. I would use Opengl for the editor GUI.

[Aurel]

I must say that i am not happy about that.
First i have new version of AurelEdit with scilexer version 2.2.0.0
which i finally reconstruct to work with my editor .
But unfortunatelly comment block /* */ is nor colored with VB -8 lexer 
line comment work.
Compile custom version is not bad idea at all .
Also Vb lexer don't have keywords2 or keywords3  coloring  as far as i can tell
or maybe i miss something.
On the end do we can agree which version of scintilla  to use? 
I mean in forst place on potencial users.
Someone may use OxIDE, which use 2.2.0,0
also Arnold O2Hedit currentlly same
and i switch to 2.2.0.0. too so when someone try editors he expect that all of them
highlite properly oxygen basic files ...
did i miss something ?

[Arnold]

Hi Aurel,

there should be no problem to use the special Scilexer.dll version 3.61 or 3.74. The only thing you have to do is change from:
.... SCI_SETLEXER, 8, 0
to
.... SCI_SETLEXER, 200,0    'Oxygenbasic lexer

200 is an arbitrary constant which I used for Oxygenbasic. You can see this in Scilexer.h (#define SCLEX_OXYGENBASIC 200) in the previous uploaded zip file.
To see what is possible you should download the source of Scintilla version 3.61 or 3.74 at Sourceforge. There is a directory doc which contains  ScintillaDoc.html whith a lot of useful information.

Roland

[Arnold]

Hi Charles,

developing OxIde with Opengl is a very interesting project. I suppose you will not need Scintilla at all in this case?
I will use the special Scilexer.dll for my little editor project as it offers some more features than version 2.20. It is a little bit modeled after the qeditor of the masm32 project, which is a masterpiece of it's own and which this editor will never reach, but I would like to add some more features to it.

Roland

[Aurel]

Hi Arnold

So conclusion is we cannot find same interest for scintilla lexer what might be good and not good.
I am really wondering what kind of special features have this newer versions than
we already have ..i really doubt that i need them.
On the other side size of 1MB for a dll is too big if you ask me .
For example with old version 1.6.8.0 i have very fine syntax coloring
,folding work,markers work even intelisense work...

[Aurel]

Arnold
I download your 361sci zip and replace 2.2.0,0 version
with this new one
setlexer = 200 as you say and yes syntax hilite work but when i try to use
block comment  /*  */ then insted of green text i get changed font 

.

[Arnold]

Charles,

after some more thoughts I think it is not really necessary to include this special dll with Oxygenbasic. This perhaps would make some sense if Oxide would stay with Scilexer.dll. The dll was a byproduct which resulted because I was interested to see how Scintilla works. (And now I noticed that I forgot to reflect about the "`" char which can also be used in Oxygenbasic for strings).
I can use the dll nevertheless with my little project. I will rename the dll and maybe go back some versions - I do not need all the features. So Aurel can continue to work with version 2.20 too.

Roland

[Aurel]

So Aurel can continue to work with version 2.20 too.

Hi Arnold
I will try to continue with this 2.2.0.0 version but it looks that VB lexer is not very good solution
maybe i must try  PowerBasic or FreeBasic lexer to see how work.
As both of you know LOUT lexer used in old AScio2 editor work very well.
Even i don't know what LOUT is  shame on me 
uh.. openGL based editor is a interesting idea BUT i don't know I simply don't have good experience
with openGL and i prefer native windows things more.
From my point of view "the best" option should be use richedit derived control to use for syntax coloring.
For example CreativeBasic use sometihng like that and coloring work like a charm BUT..
yes again BUT ,author use MFC.

[Arnold]

when i try to useblock comment  /*  */ then insted of green text i get changed font 

Hi Aurel,

it would work. I used your last AurelEdit.o2 bas (you forgot to add % WS_EX_TOPMOST=8 somewhere) and did these modifications:

''      SendMessage hsci, SCI_SETLEXER, 8, 0
SendMessage hsci, SCI_SETLEXER, 200, 0
...
      sn 6,  rgb(200,0,0)     'symbols
      sn 7,  0x002000  '

      sn 23,  0x207800  'block comment
      sn 24,  0x207800  'commentline //
...

and the comments will show.

If you check the file /include/Scintilla.iface which is included in the previous zip file you will find the location for the possible styles which can be influenced and which are then defined in Scilexer.h:

# Lexical states for SCLEX_VB, SCLEX_VBSCRIPT, SCLEX_POWERBASIC, SCLEX_BLITZBASIC, SCLEX_PUREBASIC, SCLEX_FREEBASIC, SCLEX_OXYGENBASIC
lex VB=SCLEX_VB SCE_B_
lex VBScript=SCLEX_VBSCRIPT SCE_B_
lex PowerBasic=SCLEX_POWERBASIC SCE_B_
lex BlitzBasic=SCLEX_BLITZBASIC SCE_B_
lex PureBasic=SCLEX_PUREBASIC SCE_B_
lex FreeBasic=SCLEX_FREEBASIC SCE_B_
lex OXYGENBASIC=SCLEX_OXYGENBASIC SCE_B_
val SCE_B_DEFAULT=0
val SCE_B_COMMENT=1
val SCE_B_NUMBER=2
val SCE_B_KEYWORD=3
val SCE_B_STRING=4
val SCE_B_PREPROCESSOR=5
val SCE_B_OPERATOR=6
val SCE_B_IDENTIFIER=7
val SCE_B_DATE=8
val SCE_B_STRINGEOL=9
val SCE_B_KEYWORD2=10
val SCE_B_KEYWORD3=11
val SCE_B_KEYWORD4=12
val SCE_B_CONSTANT=13
val SCE_B_ASM=14
val SCE_B_LABEL=15
val SCE_B_ERROR=16
val SCE_B_HEXNUMBER=17
val SCE_B_BINNUMBER=18
val SCE_B_COMMENTBLOCK=19
val SCE_B_DOCLINE=20
val SCE_B_DOCBLOCK=21
val SCE_B_DOCKEYWORD=22
val SCE_B_COMMENTBLOCK1=23
val SCE_B_COMMENTLINE=24

Style 23 and 24 were added by me. Scintilla version 2.20 can use style 0-18, version 3.61 can use style 0-22. The lexXXX.cxx files will then decide which styles are used and how they are used.

And although this makes Scintilla quite flexible it is also a problem. If you have special features for a language you will have to write a special lexer. And Oxygen is very special regarding comments. And it is very difficult to get all wrapped up.

E.g. there is SCB_COMMENTBLOCK=19 which is used for the commentblock of Freebasic /' ... '/. This is different from the commentblock of Oxygenbasic /* ... */ so you have to add a new routine in LexBasic.cxx and if there are more such exceptions you will have to write a lexer from scratch or you have to arrange somehow with the existing lexers.

Roland

[Aurel]

Hi Arnold 
I understand now and thanks for explanation...
because i propbably never figured what you exactly do in your custom version
So good to know that VB lexer support 0-18 words coloring scheme 
Also i dont know that FreeBasic use (if you ask me stupid way for comment block).
Most standard way is a C style comment block with  /* ....  */
and for line comment just [ ' ]
PureBasic have [ ; ] just because support FLATasm comment 
ok i will try all options ..
mine main problem is compiler path and i must find way to save path in some sort of config file
So when user run editor ,first once set path / save and that is ..right?

[Arnold]

I could not resist and investigated some more time for creating a scilexer.dll for Oxygenbasic. I found that I can reduce the size of the dll version 3.74 if I use the NO_CXX11_REGEX flag. This will only use scintilla's basic regex features but will reduce the size of the dll of about 500 kb.

So I created a special scilexer.dll version 3.74. I also used the extra lexers for Lout, PB and VB (not cpp). So for the Basic languages it should be compatible with version 2.2.0.

For use with OxygenBasic I added these values:
#define SCLEX_OXYGENBASIC 200
and the styles
SCE_B_COMMENTBLOCK1=23   (C-like block comments)
SCE_B_COMMENTLINE=24    (C-like line comment)
SCE_B_STRING_GRAVE=25   (strings in Oxygen using the ` grave accent)

I did not even try to set a style for string blocks which are possible in Oxygenbasic too. But until now I could only highlight string blocks with JFE (Jen's file editor) and only if the block was completely displayed in the window.

I also modified scintilla.inc to reflect the changes up to version 3.74. (I modified Sci_NotifyHeader a little bit, because otherwise WM_NOTIFY does not work correctly).

The dll was created with gcc version 4.8.1 and the size is 610 kb. I do not have access to Visual Studio 2015, probably the size could be reduced by another 100 kb.

Attached is the zip file with the dll, scintilla.inc and the necessary files to build the dll with mingw or Visual Studio. Maybe it can be of some help in the future.

Roland



.

[Aurel]

Arnold
Do you still use this version of scilexer ?
and how work?