Tuesday, November 29, 2005
LispOS revisited...
I'm not sure quite why, but today I was hit by a wave of nostalgia which caused me to reread some of the posts of the LispOS mailing list (archived at: tunes.org). These mailing archives are responsible for leading me to CLIM, McCLIM then DUIM (in that order) in search of a decent libre Lisp environment.
I had a few thoughts (very few of which are actually original) on what I'd like to see in a Lisp environment (if not a Lisp OS). I'd like to record them for purely personal reasons - so I'm going to.
1. Initially it would have to be based on top of an existing OS (for driver support mainly, and also so that existing (non-Lisp) apps are available). Theoretically this could be any OS (Linux, *BSD, NT, ...) as long as the OS interfaces are sufficiently abstracted.
2. The Lisp world and the native world need to interoperate nicely. For a unix interface this means basically 3 things...
2.1: The Lisp world needs to be able to make use of the available libraries on the host system (at a level of abstraction higher than FFI; lots of library wrapping).
2.2: The native processes need to be able to invoke functionality in the Lisp world direct from the shell (e.g. '> wc -l myfile.txt | some-lisp-fn > output.txt') which would require either a non-standard shell, or a hacked Bash (or whatever). Ideally from the user's point of view it shouldn't be possible to tell which commands are Lisp, and which are native. I suspect this will be easiest in a new shell, but that's unlikely to provide the potential for wide-spread usage I'd like to see.
2.3: The native side needs to be able to call into the Lisp world, as if the Lisp world were just a big shared library.
3. The chances of this kind of environment catching on in the large, in the real world, is pretty small (since most organisations have so much invested already in other technologies such as C and Java (or .net)). So I think it would need to be possible to write code in Lisp, and package that code as Java .class files or C libraries.
This needs to work both ways (i.e. be able to use .class files in the Lisp world, the C interface is already mentioned above). Mechanisms already exist to enable this interoperation but again, further abstraction would be nice.
4. Much of the initial work would appear to be writing FFI glue code or file-processing code so that tools can be developed at particular levels in the architecture (e.g. write a mail client in Lisp, but keep the underlying mail sending / receiving / queuing functionality native, or write a tool to edit passwd files that operates on abstract data (lists? ;-) but that ultimately updates the file itself).
5. It might be necessary to be able to ship the Lisp runtime (sans compiler etc.) as a shared library so software developed in the environment can be used seamlessly by users of the native environment without them having to install a full Lisp development system. (I'm not sure this is actually needed, but in terms of marketing its probably something that would be good, politically.) This would reduce the burden of supporting users in any case (i.e. they wouldn't be able to stuff things up) and Lisp people will have access to the full environment anyhow.
Certainly for small throw-away utilities (perhaps like the passwd editing tool briefly mentioned earlier) its imaginable that non-Lispers won't want a full environment (many Linux users never compile from source for example).
6. Suitable abstractions for accessing hardware devices would be nice so the devices can be accessed from Lisp (apparently directly) but the environment would still use the native drivers. DUIM is obviously the abstraction for the windowing system, but why not abstractions (better than currently available) for accessing all sorts of different devices (mouse, screen, ports, modem, graphics tablet....)?
Obviously this won't be necessary unless people want to take direct control of things at the device level.
7. Perhaps there's a way to make FASLs (more?) self-contained (i.e. instead of building FASLs based on a particular source file, build 1 FASL per package instead (or some other compilation unit)) so that Lisp 'apps' can be loaded on demand (when thinking of many OS processes talking to a single Lisp world, each conceptual 'OS application' implemented in Lisp will need to be loaded into the world) since its unlikely anybody will want to load all available 'Lisp apps' into the core at the same time. Following this logic through it seems like it might be useful to be able to *unload* the FASLs too, perhaps at GC time (for example, the Lisp version of 'wc' might be loaded at some point when the user invokes it from the shell, it probably doesn't need to be kept in core forever afterwards, especially if the user doesn't invoke it very often).
I think the above could provide a solid and useful hosted Lisp environment; once that's available, and the environment is sufficiently developed (i.e. good editor, debugger, stepper, introspection capabilities...) it should be possible to remove things from other layers of the application / OS stack - eventually leading to a full LispOS, possibly.
All that kind of fleshes out my TODO list somewhat:-
. Finish DUIM port to Lisp (I guess at least 2, maybe 3, months)
. Port DUIM extensions (presentations and the other FunO stuff)
. Port DEUCE
. Write Lisp mode for DEUCE
. Write Lisp Listener
. Implement some method for running native apps and pulling any output back into the Listener
. Decent debugger / inspector etc.
. Stuff from the above list
That should keep me going for a few years... ;-)
I had a few thoughts (very few of which are actually original) on what I'd like to see in a Lisp environment (if not a Lisp OS). I'd like to record them for purely personal reasons - so I'm going to.
1. Initially it would have to be based on top of an existing OS (for driver support mainly, and also so that existing (non-Lisp) apps are available). Theoretically this could be any OS (Linux, *BSD, NT, ...) as long as the OS interfaces are sufficiently abstracted.
2. The Lisp world and the native world need to interoperate nicely. For a unix interface this means basically 3 things...
2.1: The Lisp world needs to be able to make use of the available libraries on the host system (at a level of abstraction higher than FFI; lots of library wrapping).
2.2: The native processes need to be able to invoke functionality in the Lisp world direct from the shell (e.g. '> wc -l myfile.txt | some-lisp-fn > output.txt') which would require either a non-standard shell, or a hacked Bash (or whatever). Ideally from the user's point of view it shouldn't be possible to tell which commands are Lisp, and which are native. I suspect this will be easiest in a new shell, but that's unlikely to provide the potential for wide-spread usage I'd like to see.
2.3: The native side needs to be able to call into the Lisp world, as if the Lisp world were just a big shared library.
3. The chances of this kind of environment catching on in the large, in the real world, is pretty small (since most organisations have so much invested already in other technologies such as C and Java (or .net)). So I think it would need to be possible to write code in Lisp, and package that code as Java .class files or C libraries.
This needs to work both ways (i.e. be able to use .class files in the Lisp world, the C interface is already mentioned above). Mechanisms already exist to enable this interoperation but again, further abstraction would be nice.
4. Much of the initial work would appear to be writing FFI glue code or file-processing code so that tools can be developed at particular levels in the architecture (e.g. write a mail client in Lisp, but keep the underlying mail sending / receiving / queuing functionality native, or write a tool to edit passwd files that operates on abstract data (lists? ;-) but that ultimately updates the file itself).
5. It might be necessary to be able to ship the Lisp runtime (sans compiler etc.) as a shared library so software developed in the environment can be used seamlessly by users of the native environment without them having to install a full Lisp development system. (I'm not sure this is actually needed, but in terms of marketing its probably something that would be good, politically.) This would reduce the burden of supporting users in any case (i.e. they wouldn't be able to stuff things up) and Lisp people will have access to the full environment anyhow.
Certainly for small throw-away utilities (perhaps like the passwd editing tool briefly mentioned earlier) its imaginable that non-Lispers won't want a full environment (many Linux users never compile from source for example).
6. Suitable abstractions for accessing hardware devices would be nice so the devices can be accessed from Lisp (apparently directly) but the environment would still use the native drivers. DUIM is obviously the abstraction for the windowing system, but why not abstractions (better than currently available) for accessing all sorts of different devices (mouse, screen, ports, modem, graphics tablet....)?
Obviously this won't be necessary unless people want to take direct control of things at the device level.
7. Perhaps there's a way to make FASLs (more?) self-contained (i.e. instead of building FASLs based on a particular source file, build 1 FASL per package instead (or some other compilation unit)) so that Lisp 'apps' can be loaded on demand (when thinking of many OS processes talking to a single Lisp world, each conceptual 'OS application' implemented in Lisp will need to be loaded into the world) since its unlikely anybody will want to load all available 'Lisp apps' into the core at the same time. Following this logic through it seems like it might be useful to be able to *unload* the FASLs too, perhaps at GC time (for example, the Lisp version of 'wc' might be loaded at some point when the user invokes it from the shell, it probably doesn't need to be kept in core forever afterwards, especially if the user doesn't invoke it very often).
I think the above could provide a solid and useful hosted Lisp environment; once that's available, and the environment is sufficiently developed (i.e. good editor, debugger, stepper, introspection capabilities...) it should be possible to remove things from other layers of the application / OS stack - eventually leading to a full LispOS, possibly.
All that kind of fleshes out my TODO list somewhat:-
. Finish DUIM port to Lisp (I guess at least 2, maybe 3, months)
. Port DUIM extensions (presentations and the other FunO stuff)
. Port DEUCE
. Write Lisp mode for DEUCE
. Write Lisp Listener
. Implement some method for running native apps and pulling any output back into the Listener
. Decent debugger / inspector etc.
. Stuff from the above list
That should keep me going for a few years... ;-)
Wednesday, November 23, 2005
DUIM port progress
Time is flying by. I find it hard to believe that nearly a month has passed since my last post. I think a progress report is in order...
I've tidied the code up significantly from where it was a month ago; the consistency of the code is much improved.
I removed the DEFINE-ABSTRACT-CLASS and added DEFINE-DYLAN-CLASS instead. The latter accepts a list of modifiers (for example, (:sealed :abstract)). These modifiers don't actually do anything but it does mean the Lisp code reads more like the original Dylan.
I've added the recording module to the code base, and moved to an ASDF build which makes life a little easier.
Most of the macros are complete now (although they need a serious amount of testing which should be possible Real Soon Now(tm) (I've created the user and core modules too which should provide a convenient testing sandbox)). Only the really hard (or scary-looking, at least) macros are outstanding and I have a 90% clean compile.
Things left to do before this phase is finalised:-
1. Finish converting the hairy macros (frame definer, command table definer, etc.)
2. Clean the compile up
3. Decide on a test framework (I'm leaning towards Franz' open source testing framework at the moment, but want to look at the Dylan Testworks code too which is in the fundev source tree).
4. Write unit tests and documentation strings for the DUIM functions.
5. Port the DUIM tests from Dylan into Common Lisp and get them running too.
Once that's all done I'm back to my original plan of implementing a back end and starting on application development.
I've tidied the code up significantly from where it was a month ago; the consistency of the code is much improved.
I removed the DEFINE-ABSTRACT-CLASS and added DEFINE-DYLAN-CLASS instead. The latter accepts a list of modifiers (for example, (:sealed :abstract)). These modifiers don't actually do anything but it does mean the Lisp code reads more like the original Dylan.
I've added the recording module to the code base, and moved to an ASDF build which makes life a little easier.
Most of the macros are complete now (although they need a serious amount of testing which should be possible Real Soon Now(tm) (I've created the user and core modules too which should provide a convenient testing sandbox)). Only the really hard (or scary-looking, at least) macros are outstanding and I have a 90% clean compile.
Things left to do before this phase is finalised:-
1. Finish converting the hairy macros (frame definer, command table definer, etc.)
2. Clean the compile up
3. Decide on a test framework (I'm leaning towards Franz' open source testing framework at the moment, but want to look at the Dylan Testworks code too which is in the fundev source tree).
4. Write unit tests and documentation strings for the DUIM functions.
5. Port the DUIM tests from Dylan into Common Lisp and get them running too.
Once that's all done I'm back to my original plan of implementing a back end and starting on application development.
