ScriptBasic O2h Embedding

[JRS]

Charles,

That is excellent news!

I'm glad the SB ext. mod. is taking shape as well.

John

[JRS]

A while ago, I was trying to use BCX to create an extension module rather than use C. (traditional method) Armando gave me a hand and posted this.

John, here's the problem:

Those macros (besSub/besFunction/besVERSION_NEGOTIATE) are actually function/sub aliases.

You would be better off in the long run taking the time to translate those manually into BCX function/sub syntax.

The one for besVERSION_NEGOTIATE is an interesting one, because it aliases this when using g++:

Code: C

1. int extern "C" versmodu(int Version, char *pszVariation, void **ppModuleInternal){

I believe what is causing the error you listed above is the order of int extern "C".  I think it should be extern "C" int

Before you say "but SB compiles fine from the makefiles" remember that SB uses gcc under non-windows environments, so the extern "C" decorator is not applied. MBC and BCXU use g++.

Another option is fixing this in the header files themselves.  Flip the order of any macro that has <variable-type> DLL_EXPORT to DLL_EXPORT <variable-type> and the errors go away....

For example, with besVERSION_NEGOTIATE:

THIS

Code: [Select]

#define besVERSION_NEGOTIATE int DLL_EXPORT versmodu(int Version, char *pszVariation, void **ppModuleInternal){
BECOMES THIS:

Code: [Select]

#define besVERSION_NEGOTIATE DLL_EXPORT int versmodu(int Version, char *pszVariation, void **ppModuleInternal){
BTW, besVERSION_NEGOTIATE translated to BCX becomes:

Code: Text

1. function versmodu(Version as integer, pszVariation as string, ppModuleInternal as void ptr ptr) as integer export

So it's not really that difficult.  You wouldn't be converting ALL of the macros, just the ones that pertain to function/sub aliases....

I then used Peter Verhas's trial example extension module code to create a BCX version.

Code: [Select]

$DLL
$EXECON "-I/usr/src/scriptbasic"
 
#include <basext.h>

'besVERSION_NEGOTIATE
FUNCTION versmodu(Version as integer, pszVariation as string, ppModuleInternal as void ptr ptr) as integer export
  FUNCTION = INTERFACE_VERSION
END FUNCTION

'besSUB_START
FUNCTION bootmodu(pSt as pSupportTable, ppModuleInternal as void ptr ptr, pParameters as pFixSizeMemoryObject, pReturnValue as pFixSizeMemoryObject ptr) as integer export
  DIM pEo AS pExecuteObject
  DIM pL AS long ptr
  besMODULEPOINTER = besALLOC(sizeof(long))
  IF besMODULEPOINTER = NULL THEN FUNCTION = 0
  pL = (long *)besMODULEPOINTER
  *pL = 0
  FUNCTION = 0
END FUNCTION
 
'besSUB_FINISH
FUNCTION finimodu(pSt as pSupportTable, ppModuleInternal as void ptr ptr, pParameters as pFixSizeMemoryObject, pReturnValue as pFixSizeMemoryObject ptr) as integer export
  DIM pEo AS pExecuteObject
  FUNCTION = 0
END FUNCTION

'besFUNCTION
FUNCTION trial(pSt as pSupportTable, ppModuleInternal as void ptr ptr, pParameters as pFixSizeMemoryObject, pReturnValue as pFixSizeMemoryObject ptr) as integer export

  DIM pEo AS pExecuteObject
  DIM pL AS long ptr

  PRINT "Function trial was started..."
  pL = (long *)besMODULEPOINTER
  (*pL)++
  besRETURNVALUE = besNEWMORTALLONG
  LONGVALUE(besRETURNVALUE) = *pL
 
  PRINT "Module directory is ";(char*)besCONFIG("module")
  PRINT "dll extension is ";(char*)besCONFIG("dll")
  PRINT "include directory is ";(char*)besCONFIG("include")

  FUNCTION = 0
END FUNCTION


[Charles Pegge]

Hi John,

I've now got the module interface working. I found Assembler was the easiest way to do it. This is my workbench code for testing various data types.

Because it is not using the macros and a C compiler, we have to trust in the stability of the data structures and function tables across different builds.

Do you normally recompile all the extension modules when the core SB is recompiled?

Code: [Select]

'extension module for ScriptBasic:

  '23:02 11/05/2011
  '23:31 13/05/2011
  '
  'Charles Pegge

  #file "mdlt.dll"


  '----------------------------------------------------------------------------------
  function versmodu cdecl (sys Version, pszVariation, ppModuleInternal) as sys export
  '==================================================================================
  'print "Version: " hex version
  return Version
  end function


  '-------------------------------------------------------------------------------------------
  function bootmodu cdecl (sys pSt, ppModuleInternal, pParameters, pReturnValue) as sys export
  '===========================================================================================
  'print "Boot!"
  end function



  '------------------------------------------------------------------------------------
  function finimodu cdecl (sys pSt, ppModuleInternal, pParameters, pReturnValue) as sys export
  '====================================================================================
  'print "Finish!"
  end function




  '----------------------------------------------------------------------------------------
  function trial cdecl (sys pSt, ppModuleInternal, pParameters, pReturnValue) as sys export
  '========================================================================================
  sys p,t,v,w,pm,pn
  string s
  '
  if pParameters=0 then
    print "no params"
  else
    '
    'HOW MANY ARGUMENTS GIVEN
    '------------------------
    '
    mov ecx,pparameters
    mov eax,[ecx+8] : shr eax,2 : mov w,eax
    'mov eax,[ecx+12] : mov t,al
    '
    'print "No of Arguments given: " w
    '
    'SELECT PARAM pn
    '---------------
    '
    'pParameters->Value.aValue[(X)-1]
    '
    pn=1
    '
    mov ecx,pparameters
    mov ecx,[ecx] 'array of param pointers
    mov eax,pn : dec eax : shl eax,2
    mov ecx,[ecx+eax] 'array offset to get pparam
    '
    mov pm,ecx
    '
    'CONVERSION TO STRING
    '--------------------
    '
    'to string f4 / to long f8 / to double fc
    'pSt->Convert2String(pSt->pEo,(x),pSt->pEo->pGlobalMortalList)
    '
    mov ecx,pSt : mov eax,[ecx] : push [eax+0x8c] : push pm : push eax : call [ecx+0xf4]
    mov ecx,eax
    mov p,ecx
    zstring* z : &z=*p
    '
    'print "As String:  " z
    '
    mov ecx,pSt : mov eax,[ecx] : push [eax+0x8c] : push pm : push eax : call [ecx+0xf8]
    mov ecx,[eax]
    mov p,ecx
    '
    'print "As Long:  " p
    '
    mov ecx,pSt : mov eax,[ecx] : push [eax+0x8c] : push pm : push eax : call [ecx+0xfc]
    mov ecx,eax
    mov p,ecx
    double*d : &d=p
    '
    print "As Double:  " d
    '
    'GET VALUE / STRING POINTER
    '--------------------------
    '
    'mov eax,[ecx]    : mov p,eax  'value / pointer
    'mov eax,[ecx+8]  : mov w,eax  'size
    'mov eax,[ecx+12] : and eax,-1 : mov t,eax
    '
    'type codes 0 undef 1 double 2 long 3 zstring* 4 zchar ff ptr
    '
  end if
  '
  '
  '============
  'RETURN VALUE
  '============
  '
  '
  'check stack:
  'mov w,esp
  '
  '
  ' 00c string  010 long  018 double
  '---------------------------------
  '
  '
  'RETURN LONG
  '-----------
  '
  'long v=456
  '
  'mov ecx,pst : mov eax,[ecx] : push [eax+0x8c] : push [eax+0x94] : call [ecx+0x10]
  'mov ecx,pReturnValue : mov [ecx],eax : mov edx,v : mov [eax],edx
  '
  '
  'RETURN DOUBLE
  '-------------
  '
  'double d=1234.5678
  '
  'mov ecx,pst : mov eax,[ecx] : push [eax+0x8c] : push [eax+0x94] : call [ecx+0x18]
  'mov ecx,pReturnValue : mov [ecx],eax : fld qword d : fstp qword [eax]
  '
  '
  'RETURN STRING
  '-------------
  '
  long ls=8 'length excluding null terminators
  zstring z at p
  '
  mov ecx,pst : mov eax,[ecx] : push [eax+0x8c] : push ls : push [eax+0x94] : call [ecx+0x0c]
  mov ecx,pReturnValue : mov [ecx],eax : mov eax,[eax] : mov p,eax
  z="ABCDEFGH" 'direct input
  '
  '
  'NB: beware cdecl cleanup.
  '-------------------------
  '
  'sub w,esp
  'print "check stack: " hex w
  '
  end function



Charles

[JRS]

ASM is the ultimate glue language. I don't think you will find an assembly level programmer that will say it can't be done.

ScriptBasic is unique in that it builds everything from scratch from the source when you run setup. (.h, syntax tables, help, ...) A custom configure program might be the best way to explain it. On the Linux side, Armando built a standard makefile to compile the source without the rebuild.

SB Makefile

You can build a module using the setup --module=O2 or just run the make for the module in it's extension directory.

[JRS]

I compiled your framework SB extension module and this is what it returned for me. (Ubuntu 11.04 64 bit - Wine 1.3)

Code: [Select]

declare sub trial alias "trial" lib "mdlt"

rtn = trial(1)
PRINT rtn,"\n"

rtn = trial(1.1)
PRINT rtn,"\n"

rtn = trial("One")
PRINT rtn,"\n"

The following O2 print dialogs are in the order they occurred.

Code: [Select]

C:\o2h\a33\examples\SBO2>scriba trial.sb
ABCDEFGH
ABCDEFGH
ABCDEFGH

C:\o2h\a33\examples\SBO2>


[JRS]

Charles,

After working with your O2 extension module framework for SB, I realized that what you posted is a test suite rather than a O2 version to the trial example Peter Verhas did to help developers understand how to create an extension module.

This example shows passing the variable types SB supports. (long, double and strings) I only saw one commented line of code that looks like you were creating a new SB variable within O2. I don't want to side track you from the GUI work your doing with a O2 thread. When you are ready, I think a SQLite3 extension module in O2 may be a worth while project that would fill a void.

John

[Charles Pegge]

Yes John I started with 'trial' and morphed into a set of i/o tests. I found a small misalignment of function table offsets -1 slot less than I thought it should be according to the headers. So I initially got longs instead of string parameters. I also found that all the calls were cdecl, not stdcall, even though we are in an ms environment.

I also looked at the possibility of passing variables Byref but apart from zStrings I have not seen any provision for it, not in the macros anyway.

Otherwise a smooth ride so far

Charles

[JRS]

You may want to look at Peter's MyAlloc library. It is used in SB and makes memory/string management easier to work with especially in a multi-threaded environment. Peter is a talented programmer that developed an amazing Basic Scripting API that is like a Christmas present I still haven't finish unwrapping.

This module is a general purpose memory allocation module, which can be used in any project that needs heavy and sophisticated memory allocation. Originally the module was developed for the ScriptBasic project. Later we used it for Index.hu Rt AdEgine project and multi-thread features were introduced.

ScriptBasic is fast, it generates a compact internal code, which is interpreted. This internal code occupies a single, continuous memory chunk and is usually saved into a cache file. The cache file is automatically checked by ScriptBasic and thus it compiles the source only when needed.

[JRS]

I think SB was initially developed on Linux based on the ANSI/ISO C standard. Looking at the SB source, you can see that WIN32 was patched in afterwards with #define's. I find the history behind calling conventions interesting.

When compiling for the x64 architecture using Microsoft tools, there is only one calling convention — the one described here, so that stdcall, thiscall, cdecl, fastcall, etc., are now all one and the same.

Can you hear the sound of another door closing?

[Charles Pegge]

Its the same for Linux/Mac 64. there is only one standard calling convention but it differs from the MS standard. I'm not so keen on passing floating point params in the XMM registers.

I found an excellent document describing all the standards across different platforms and compilers:

http://www.agner.org/optimize/calling_conventions.pdf

Charles

[JRS]

I downloaded the O2 in-process zip and found some new demos. This is the first example of a SBO2 GUI.

sbo2demo9.sb

Code: [Select]


  ' ======================================
  ' WORKING WITH A PERSISTENT O2H PROGRAME
  ' ======================================

  '09:08 13/05/2011

  declare sub dll alias "dyc" lib "dyc"

  ' ==================
  ' OXYGEN SOURCE CODE
  ' ==================
  '
  O2prog="""
  '
  $ THREADED_WINDOW
  '$ PERSISTANT_WINDOW
  include "GraphWin1B.inc"
  '
  '
  function Commune(zstring * SBSends, zstring * SBReceives, sys bufferlength) as sys external
  '------------------------------------------------------------------------------------------
  'print SBsends
  string tab=chr 9, cr=chr(13)+chr(10)
  single v=(1 2 3 4)*2+1.25
  SBReceives= SBSends cr+
  "O2H responds" tab str(v)  tab +
  "Enter x,y coordinates:" cr+
  "or empty line to quit" tab chr(0)
  end function
  '
  '
  function SetPosition(long x, long y) as long external
  '----------------------------------------------------
  'print "x: " x "  y: " y
  while dat
    if active then sleep(0) else return -1
  wend
  dat[11]<=x,y
  dat=1 'signal update
  return active
  end function
  '
  '
  function CloseWindow() as sys external
  '-------------------------------------
  while dat
    if active then sleep(0) else return -1
  wend
  dat=2 'request shutdown
  return active
  end function
  '
  '
  function IsActive() as sys external
  '----------------------------------
  sleep 0
  return active
  end function

  map <= @Commune, @SetPosition, @CloseWindow, @IsActive

  """ & chr(0)


  ' ================
  ' SCRIPTBASIC CODE
  ' ================
 
  'INTERFACE
  '---------

  Commune    =1
  SetPosition=2
  CloseWindow=3
  IsActive   =4
  send="SB Sends a message" & chr(0)
  bufferlength = 1000
  receive = space(bufferlength)

  '---------------------------------------------------------------
  'COMPILE --> ERROR CHECK --> STARTUP --> FUNCTION CALLS --> STOP
  '---------------------------------------------------------------

  'COMPILE
  '-------

  compilerrl= dll( "ms,i,sbo2.dll,compile,ZZL", O2prog, receive, bufferlength)
  '
  'ERROR CHECK
  '-----------
  '
  if compilerrl>0 then
    print left(receive,compilerrl)
    line input w
  else
    '
    'START DEMO
    '----------
    '
    dll ("ms,i,sbo2.dll,start,L",0)
    '
    'TEST TEXT IO COMMUNICATION
    '--------------------------
    '
    dll ("ms,i,sbo2.dll,callfun,LzZL", Commune, send, receive, bufferlength)
    '
    split receive by "\t" to s1, s2, s3
    print s1,"\n"
    print format("%6.4f",s2),"\n"
    print s3,"\n"
    '
    'WAIT TILL WINDOW IS ACTIVE
    '--------------------------
    '
    ra=0
    c=0
    while ra=0
      ra=dll ("ms,i,sbo2.dll,callflex,LL", IsActive,0)
      c+=1
      if c>100 then
        ra=-1
      end if
    wend
    '
    if ra>0 then
      '
      'TEST USER COMMUNICATION WITH WINDOW
      '-----------------------------------
      '
      'enter x and y coords or quit
      '
      w=" "
      re=1
      while (asc(w)>=32) and (re>0)
        line input w
        if len(w)>1 then
          split w by "," to x,y
          re=dll ("ms,i,sbo2.dll,callflex,LLLL", SetPosition,2,x,y)
        end if
      wend
      '
      'CLOSE DOWN WINDOW
      '-----------------
      '
      dll ("ms,i,sbo2.dll,callflex,LL", CloseWindow,0)
      '
    end if
    '
    'STOP THE PERSISTANT PROGRAM
    '---------------------------
    '
    dll ("ms,i,sbo2.dll,stop,L", 0)
    '
  end if


[JRS]

Hi Charles,

I have been playing around with your mdlt.dll test SB extension module. It seems I'm able to pass strings, integers and floating point values. I noticed that you can return a string as well. Are you using the SB memory management API so if SB closes or errors, all memory is release via the SB MyAlloc version of Peter's library? Is the only way to access the SB API from the O2 extension module via ASM?

John

[Charles Pegge]

Hi John,

The interface uses the procedures encoded in the macros, and anything created for SB is "mortal". If there is an error. SB will clean up these elements. The Oxygen compiler dll will be released and shut down in an orderly way but the running O2 program may leave some debris behind when terminated prematurely. Since this is all in-process the OS should clear up on termination, but further thought is needed about how to catch any fatal run-time errors.

Assembler is not the only way to access the SB API but it very efficient and does not rely on a full set of headers. I can make it  less cryptic and maybe create a mini header.

Charles

PS: I found my alignment error, so my mapping of the API is now in agreement with the headers.

[JRS]

PS: I found my alignment error, so my mapping of the API is now in agreement with the headers.

Good thing, I would hate to have to send Peter Verhas a pink slip. 

[JRS]

Charles,

I'm trying to get an idea where things are with the O2 extension module and the goals you have with it. I think I could create a SQLite3 O2 extension module with what you have now. Can O2 access the SB scripts variables at this time? (integers, strings and arrays)

I would like to get the SQLite example I posted previously by scripting the API translated to an O2 extension module. I want to use the same syntax all database interfaces use at the script level. (MySQL, ODBC, PostgreSQL, ...)

• RealConnect
• Query
• FetchArray
• Close

I also understand your working on a Windows GUI threaded task to deal with Windows messages and provide a passive event loop to SB. Will this GUI interface also allow us to create controls and access their methods and properties in a easy to use way at the SB scripting level?

Not to forget why we came to the party, the ability to pass O2 scripts from SB and run as a JIT compiled program/shared object.

John