Proposal of a (mini) TestSuite

[Arnold]

Hi Charles,

you certainly have your own tools to check OxygenBasic's functionality. So the purpose of this little project is not to provide a full-fledged testsuite but to point to side effects which can arise if you add new features to Oxygen. The application can be used outside of Oxygen (e.g. with an usb-stick), but could also be added to Oxygen in the project's section.

The directory of the project contains these files:

OxygenBasicPath.txt: the path of OxygenBasic. If I change e.g. to OxygenBasicA040 I can create reports for different installations.
OxygenCompiler.txt: this holds either gxo2.exe or co2.exe
Test_.bas: this is a template for building a test case.
TestSuite.bat: this file starts the test cases.

Utility: this folder holds the helper progs for TestSuite.bat and the include file for the testcase.o2bas.

TestCases: here are the files with the test cases.

Reports: this folder contains the reports for the test cases.

After running the testfiles a ReportSummary.txt will be created in the main folder. This report is a bit nasty and unfair as it only shows the things which do not work.

This is the very simple logic of a test case: (short) description of the testcase, Result value, Expected Value, Expected condition (true or false).

TestSuite.bat tries to compile a testfile. If this works it tries to run the compiled exe. If there is no exe file it tries to run the file with gxo2.exe (co2.exe). In the folder Reports a report for each file is created if possible. In the main directory a ReportSummary.txt shows the errors which have occured.

As I do not know if you see some value in this project or if you would like to do some things differently, I have only added some experimental cases. But the generated ReportSummary.txt already reveals some interesting aspects:

SBwin.o2bas is no testcase. I used it because I know that it only will work with ScriptBasic. In Windows Vista I can close the program directly. In Win7 64bit I can use taskmanager or close the second console and answer 'N' at the prompt for exiting the batch file. Unfortunately I have not yet found a way to kill SBwin.exe directly in the batch file using taskkill. Perhaps start.exe or a freeware program could be used for this purpose?

Test_Constructs2.o2bas and Test_oop_feature.o2bas: These files cannot be compiled with A41. It works with A40 althought there is a message for Test_constructs2.o2bas which I do not understand (8.5 is the value if I run examples\constructs\LetObjArrays.o2bas with A040). Maybe something is wrong with my utility\minitest.inc?

The other results show wrong build test cases.

If you can use this project in some way I could provide you with some more testcases, which at the moment are in fact demos of OxygenBasic. I assume that 40 or 50 files would give you assurance that 99% will work correctly.


Roland
 





.

[Arnold]

Appended are the notable error messages for the latest build of Oxygenbasic. I assume that with some testcases you would have realized the side effects earlier?


.

[Charles Pegge]

Thanks Roland,

Some fallout from the new operator overloading facility. I anticipate it will be the last of the major feature updates!

I've spent some time thinking about what unit tests to use. Ideally, every rule deployed in the compiler should have a test associated with it. Such tests could be aggregated into several programs. This would be much more thorough than relying on 'canary' demos, and establish a set protocol when introducing new features, or optimizations.

[Arnold]

Hi Charles,

I consider this little project as a learning exercise in OxygenBasic. (I am trying some more currently). As long as I cannot achieve that it will run completely automatically it will not be of much use anyway. I am not even sure if I used the way of overloading functions in minitest.inc correctly. At least I learned how to remove Nul characters from a buffer when using getfile - if I did it the correct way. My mini thesis would be that with some spot tests it should be possible to catch some unexpected side effects.

But of course you are right. A rule based testing system is the superior method to keep stability in OxygenBasic. I searched in Internet for this topic and it is amazing what some people can achieve. Probably you permantly have to check the very flexible grammar of OxygenBasic with syntax and sementics and you have to monitor the produced intermediate and final assembly code for valid statements. As an end-user of OxygenBasic I am only interested what I can do with the language and how I can use it. The demos are really invaluable for this purpose.

Regarding the grammar: I learned that computer languages can be noted in Backus Naur Form. Could the source code of OxygenBasic be used to derive a BNF notation or does such a notation already exist? Perhaps this is really a silly question.

Roland

[Charles Pegge]

Hi Roland,

I'm glad you have found the examples useful. OxygenBasic has been developed, using code-challenges in the form of examples, rather than a set of abstract rules. So I am not sure BNF would be useful at this stage of development. Maybe when things have solidified a bit more.

What I find interesting is that a number a repeat coding patterns begin to emerge in the compiler source, which can be exploited to keep it all compact.

The hi-level aspect of the languages splits down into 26 internal meta-types (currently), with their own gramatic variations, as follows:


  " >0   compund variable,"_
  "  0   primitive variable or constant,"_
  " -1   def macro,"_
  " -2   enum enumeration, "_
  " -3   type / class,"_
  " -4   o2 intrinsic function,"_
  " -5   proc sub or function,"_
  " -6   imported sub or function,"_
  " -7   vectored sub or function - declare ... AT v,"_
  " -8   struct from typedef,"_
  " -9   reserved word,"_
  " -10  x86 register,"_
  " -11  attributes,"_
  " -12  (undefined),"_
  " -13  o2 major keywords,"_
  " -14  o2h major keywords,"_
  " -15  C type keywords,"_
  " -16  intrinsic functions,"_
  " -17  intrinsic float macro,"_
  " -18  labels,"_
  " -19  termination macro,"_
  " -20  high level macro,"_
  " -21  literal number,"_
  " -22  literal string,"_
  " -23  unparsed declaration,"_
  " -24  referenced declaration,"_


This means the compiler maintains ~26 different types of record, from which to generate code