Author Topic: Token engine concept  (Read 8768 times)

0 Members and 1 Guest are viewing this topic.

Charles Pegge

  • Guest
Token engine concept
« on: May 18, 2014, 06:20:58 AM »

Code: [Select]

'FOR READING SIMPLE SPACED WORDS

function Getword(string s,sys *i,*lw) as string
===============================================
sys ps = strptr s
byte b at (i-1+ps)
sys bi,ei
do
  select b
  case 0         : exit do
  case 33 to 255 : exit do
  end select
  @b++
enddo
bi=@b
do
  select b
  case 0       : exit do
  case 1 to 32 : exit do
  end select
  @b++
enddo
ei=@b
lw=ei-bi
i=ei-ps+1
if lw then return mid s,bi-ps+1,lw
end function



string tokerr
sys    ert
string cr=chr(13,10)


function tokenise(string s) as string
=====================================
sys aw
sys lw
sys iw=1
sys it=1
string wr
string tok
do
  wr=lcase getword s,iw,lw
  if wr="" then exit do
  if it>len tok then tok+=nuls 16000
  aw=asc wr
  select aw 'simple lookup
  case "a"
    if wr="any"
      mid tok,it,"A" : it++
    else
      ert=1
    end if
  case "b"
    if wr="brave"
      mid tok,it,"B" : it++
    else
      ert=1
    end if
  case "c"
    if wr="cat"
      mid tok,it,"C" : it++
    else
      ert=1
    end if
  case else
    ert=1
  end select
  if ert=1 then tokerr += "unknown " wr cr
  ert=0
end do
return left tok,it-1
end function

function exec(string toks)
==========================

  'JUMP BY TOKEN

  sys t[256] : t[65]=>{@FAA,@FBB,@FCC}
  '
  sys    e   = len(toks)
  sys    i   = 0
  byte  *b   = strptr(toks)-1
  sys    eb  = @b+e

  NextItem:

  @b++
  if @b>eb then exit function
  i=b : goto t[i]
  'goto t[b]
  FAA:
  print "AA"
  goto NextItem

  FBB:
  print "BB"
  goto NextItem
  '
  FCC:
  print "CC"
  goto NextItem

end function

string toks=tokenise " any brave cat "

if tokerr then
  print tokerr
else
  exec toks
end if


Aurel

  • Guest
Re: Token engine concept
« Reply #1 on: May 18, 2014, 09:33:25 AM »
Thanks Charles .  :)
Charles...
Do you can present bytecode engine concept ?
I mean something simple to understand ..

Peter

  • Guest
Re: Token engine concept
« Reply #2 on: May 18, 2014, 10:53:18 AM »
Quote
Do you can present bytecode engine concept ?

What about a storybook:  byte prince married the 64bit princess!
They are living happily in a 64k rom for the rest of their live.    ;D


Mike Lobanovsky

  • Guest
Re: Token engine concept
« Reply #3 on: May 18, 2014, 01:03:30 PM »

Aurel

  • Guest
Re: Token engine concept
« Reply #4 on: May 18, 2014, 01:08:33 PM »
Peter...Better stick to your gold coin... :P here  -> (Y)
so what is wrong with my question now ?
« Last Edit: May 18, 2014, 01:23:40 PM by Aurel »

Frankolinox

  • Guest
Re: Token engine concept
« Reply #5 on: May 20, 2014, 03:33:26 AM »
@charles: "token engine concept": I have an error at line 82 (with an array (i) with your example above, cannot run this example.

regards, frank

jack

  • Guest
Re: Token engine concept
« Reply #6 on: May 20, 2014, 05:06:10 PM »
Thanks Charles .  :)
Charles...
Do you can present bytecode engine concept ?
I mean something simple to understand ..
hello Aurel
if I were you, I would look at the USCD Pascal implementation, here's some info http://en.wikipedia.org/wiki/UCSD_Pascal

Charles Pegge

  • Guest
Re: Token engine concept
« Reply #7 on: May 20, 2014, 10:11:44 PM »
Aurel,

This example simply demonstrates converting words into tokens, then using the  tokens to execute code. It minimises string processing by using stretchable buffer tok a byte array overlay b. Mid is used to patch bytes into the the buffer, without creating any additional pieces of string.


Frank,

Try copying again. Line 82 is a blank line, and line 81 is a comment :)


Jack,

I remember Apple Pascal. One of my friends earned a good living developing a bespoke database working on the Apple2c. Compiling a module on his system to P-code usually took about 20 minutes! It kept 2 floppy disk drives very busy. The 8 bit 6502 processor would have about 1/10000 the performance of a Pentium at best.


« Last Edit: May 20, 2014, 10:26:22 PM by Charles Pegge »

Frankolinox

  • Guest
Re: Token engine concept
« Reply #8 on: May 21, 2014, 05:58:13 AM »
strange, this line doesn't work for me and produce this error message in my last picture I've attached:

Code: [Select]
i=b : goto t[i]
?

regards, frank

Mike Lobanovsky

  • Guest
Re: Token engine concept
« Reply #9 on: May 21, 2014, 06:55:30 AM »
Hello Frank,

Sorry but you haven't attached anything to your message.

"Jump by array" is a new language feature. Make sure to download the latest O2 package from the Wizard at the top of this page. :)

Aurel

  • Guest
Re: Token engine concept
« Reply #10 on: May 21, 2014, 12:05:39 PM »
Frank..frank...frank
I ask you question on ox ide topic ..but you not respond ... :o
maaaan...i really sometimes cannot figured you at all  ???
please don't get me wrong  :D

Aurel

  • Guest
Re: Token engine concept
« Reply #11 on: May 21, 2014, 12:19:24 PM »
Quote
Aurel,

This example simply demonstrates converting words into tokens, then using the  tokens to execute code. It minimises string processing by using stretchable buffer tok a byte array overlay b. Mid is used to patch bytes into the the buffer, without creating any additional pieces of string.

Charles
I will try to study this example...
But what is with error...
in source code i found few enddo mistakes...looks trivial but
i am not sure that this is the best way or easiest..

Jack
hmmm pascal OS....
i found this interesting:

1.Token threading
2.Huffman threading
3.Lesser used threading



.
« Last Edit: May 21, 2014, 02:11:06 PM by Aurel »

Charles Pegge

  • Guest
Re: Token engine concept
« Reply #12 on: May 21, 2014, 01:53:08 PM »
Hi Aurel

The error is a little below your screen shot: goto t[ i]

The earlier syntax was something like : j=t[ i] : jmp j


You need a more recent o2 to support array goto.

http://www.oxygenbasic.org/o2zips/Oxygen.zip

Aurel

  • Guest
Re: Token engine concept
« Reply #13 on: May 21, 2014, 02:13:20 PM »
Ok ...i see
Uf i forget to build new folder with new dll and supported changes
my fault  :-\

Frankolinox

  • Guest
Re: Token engine concept
« Reply #14 on: May 22, 2014, 03:24:26 AM »
well, the "token engine" example works now, after I haved installed last oxygen.dll (update 02 latest build) and the array problem is gone. Sometimes I have problems with oxygen examples from oxygen folders and after having installed newer oxygen updated versions it's gone. in general: sorry, but I am wondering if one example is working fine some month without any problems and then it's producing errors. I am looking for the reason why this happened and cannot find solutions for it. for me it's better to ask here or show problems because charles often changes programming code functions or declarations or added new macros and so on.. :)

@aurel: what question do you have asked about oxide? I am not every day present and I have a lot of troubles about my e-mail accont last week (was hacked) and much more other thingies around my compiler stuff and of course the hot wheather outside .. :) and of course there's not enough time beside my mainly job for programming

thanks for feedback.mike, charles.

nice day, frank
ps: I'v sent wrong picture this afternoon, scusi senori, but now the example works correct , that was important :)
« Last Edit: May 22, 2014, 08:49:36 AM by Frankolinox »