Hi Jack,
The COGLApp class that's the core of FBSL OpenGL renderer applications uses Windows specific routines to initialize window pixel formats and 3D canvases that rely on the native drivers' extended feature sets. They can't always be (and in this particular case aren't) emulated in a virtual machine that would use macOS' (?) own OpenGL driver directly via hardware virtualization.
MSAA stands for
multi
sample
anti
aliasing, a HW-assisted technique to eliminate edge jaggies and excessive distant texture blur. Evidently your VM somehow fails at emulating this Windows native option.
Once a similar warning is issued by COGLApp, further behavior of the program is undefined.
compiled with FB x86 the 640 x 360 view gives about 70 to 80 fps
That looks nice. The FB thread you gave the links to doesn't seem to show the OpenGL initialization code so I can't tell how platform specific it may be. At any rate, if you can compile it directly on macOS then it should run faster than it would in the VM. Next, your particular GPU may be somewhat stronger than mine, and that could also add a couple dozen FPS to the render speed. Then, my renderer is set (sync'ed with the monitor retrace frequency) to yield 60FPS max to protect the GPU from possible overload. And finally, my FBSL code is executed in interpretative BASIC rather than direct machine code like FB.
Can you attach a screenshot of your 640x360 px OpenGL window?