Industry musings

Change of blog focus

2011-05-14T11:16:00.002Z

Since I started a new blog for the work I do from time to time on DUIM, I've decided to repurpose this blog to keep notes on things I encounter professionally. For most Lispers that is going to make this blog very uninteresting indeed, since the technology I use mainly at work is Java based.

I've been developing in Java since version 1.0 if I remember correctly (self-taught from the brown nutshell book with the guys doing semaphore on the front). Since then I've developed distributed Corba components, AWT (pre Swing!) and Swing applications, Web and eCommerce solutions, several enterprise applications and tools, through to my current role developing a Knowledge Management solution, all in Java.

Finally sorted out my blogger ownership...

2011-05-14T10:43:00.003Z

... so I can update this older blog with the current address of my (ahem) newer blog focussing on DUIM. Find it at http://cl-duim.blogspot.com.

More testing progress ...

2007-02-05T17:22:00.000Z

... but not much. This is more a hint to remind me of things to be careful of than anything else.

The test results are up to:

Ran 7125 checks in 110 tests from 14 suites.
42.857143% (6 / 14) suites executed correctly.
70.90909% (78 / 110) tests executed correctly.
92.98246% (6625 / 7125) checks executed correctly.

I have no idea where the additional tests (up from 7080) have come from...

Most of the additional passes have come from changing 'CHECK-EQUAL' tests (which in the Dylan made use (I presume) of Dylan's generic '=' function, but which in the Lisp version invokes 'EQUAL' instead) to 'CHECK' and passing #'dylan-= as the functional argument.

At least now colours and space-requirement objects etc. compare as equivalent when they should (which they often didn't under EQUAL).

Next I'm going to look at all the scrolling and layout failures and hopefully knock some of those (most?) on the head.

91%+ of tests passing...

2007-02-05T14:33:00.000Z

I found a problem in the test harness code (expected-column-named-width wasn't returning a value!) which has improved the test results a little. The results are now:

Ran 7080 checks in 110 tests from 14 suites.
21.428572% (3 / 14) suites executed correctly.
64.545456% (71 / 110) tests executed correctly.
91.77966% (6498 / 7080) checks executed correctly.

The main culprits accounting for the failing tests are (approx values):

LTRB tests -- 60 fails

Layout tests -- 130 fails

Gadget tests -- 100 fails

Scrolling tests -- 275 fails

The other 15 or so fails are spread over various tests. Obviously I'm going to focus my attention on the 'big 4' mentioned above. I think when I get to 95% passing I will probably move on to focus on porting a back end properly. That leaves around 230 defects that need resolving before I can move on.

Tests Building...

2007-02-03T22:56:00.000Z

... and, for the most part, running. The count at the moment is as follows:


    Ran 7080 checks in 110 tests from 14 suites.
        21.428572% (3 / 14) suites executed correctly.
        64.545456% (71 / 110) tests executed correctly.
        88.418076% (6260 / 7080) checks executed correctly.

A couple of the tests originally present have been commented out (but only 4-5), so the figure represented above isn't quite fully-representative but I think 88% isn't bad for a first run.

Most of the failures are related to the scrolling and layout tests; hopefully there's some low-hanging fruit that will improve the figures easily to be found in those areas.

It's also worth noting that the current tests miss out some of the DUIM functionality; but I'm hopeful that once any easy fixes are put in, the port should be ready for a back end at last.

Still going...

2007-01-31T18:37:00.000Z

Well, it's been a while. I'm still working on DUIM, on and off. I'm going through an 'on' period at the moment, for better or worse :-)

I've 'fixed' some of the issues I was having with the code; in particular I've implemented a work-around for the Dylan methods that specialise on subclass(<class>) using metaclasses. Basically each class defined via define-dylan-class also gets its own metaclass defined (so the metaclass hierarchy mirrors the 'concrete' class hierarchy).

I don't like this approach :-)

In the gadget-text-parser and gadget-value-printer methods I've made use of class-prototype objects instead which is a slight improvement.

I think the best solution will be to work out how to create new CLOS specializer classes; implementations provide EQL specializers and I guess it should be possible to implement a SUBCLASS specializer. I'll have to look into this.

The code is still confused regarding the use of lists and vectors, but on the whole the codebase at least hangs together in this regard. This needs revisiting at some point in the future. It might be nice to implement (or find, it's possible something exists already) a generic CLOS-based collections package (I can see the value of operating on lists / arrays via different methods, for efficiency reasons -- but I really like the Dylan approach that wraps all collection types up in a single API).

I've stopped working on any back-end code at the moment. My priority is in getting the fun-O tests to run (not necessarily pass!). At the moment around 2k of them run, and about 85% pass so I'm nearly there with this, I think. It does seem that some of the tests target a different version of DUIM than the code I have though; the most obvious case of this is in the tests for text-styles, which don't pass a name to the methods. All the text-style code in DUIM appears to require a name (as well as a family). The documentation I've seen supports the tests, but the documentation might be out of date. There are other examples too (<TEST-STATUS-BAR-PANE> inherits from <LEAF-PANE> but the status-bar handling methods want to (setf (sheet-children bar) ...); this works in DUIM since DUIM's status bar inherits from <ROW-LAYOUT> which includes <MULTIPLE-CHILD-MIXIN> -- but if the DUIM code was always organised the way it is at the moment, the status bar tests can never have worked.

I guess I'll get there in the end.

My plan of attack at the moment is:

Get the fun-o tests running, and fix any low-hanging fruit to get as many passing as possible. At this point, DUIM-CORE, DUIM-TESTS and DUIM-EXTENDED will be completely ported, barring errors (I don't think there are tests for the extensions for example)

Write a back-end. I'll probably port the existing win32 back end since at the moment that's the platform on which I'm doing most development on this. I want to see something happening on-screen now...

Port the examples and demos -- and write a couple more

Port DEUCE

I think there may be some light appearing at the end of the tunnel :-)

Current status

2006-06-25T14:42:00.000Z

Ok, once again I've been slacking rather when it comes to updating this blog. Apologies for watchers, but I have a bit of an excuse this time -- namely that my family has been expanded by a new arrival, Isaac Arthur Rose, who was born at 7.15 am on the 10th of June. Needless to say I've been a little distracted with that.

There has been some progress with DUIM. I initially decided to write unit tests using the RT framework (which I did, up to a point). Over the last couple of weeks I've converted the FunDev tests to Lisp, and written a very simple equivalent of FunDev's 'TestWorks' harness to reduce the changes to the FunDev tests that needed to be made. Since there were a couple of problems using RT for me (difficult to test intermediate states, since a bunch of the DUIM code isn't functional) I figured I'd use this quick hack for all my tests, and have now also moved my unit tests over to this new framework.
Currently, there's a bunch of code that fails the tests (and the converted Dylan tests don't yet compile) but I feel I'm making (good) progress at the moment.

Stuff to do:
o My unit-test framework is very bad at running multiple suites, and could do a better job of reporting its results. I need to give it a little polish, document it, and then perhaps release it on its own (oh joy, yet another test framework!). It also needs to do better at handling errors in the test setup / teardown code (this work is currently done outside the framework altogether, so failing code kills the test run which is not ideal.
o I need to formalise (and document) a strategy for object creation within DUIM. Currently I'm specialising MAKE-INSTANCE by EQL specialising on STANDARD-CLASS which may or may not be legal (there was a thread on c.l.l. about this recently, dig out the reference). Note also that DUIM occasionally has a MAKE method for subclasses of some type, rather than for specific types. The Dylan tests mentioned above also do this, so I need to come up with something reasonable in the near future.
o The code is currently quite schizophrenic over its handling of vectors and lists. Dylan is more forgiving than Lisp when it comes to dealing with these types (for example, it appears to be legal to give APPLY a vector as its final argument) and parts of the DUIM code rely on this. The transliteration currently uses the same types as the original Dylan code so I've hit a few places where vectors need to be converted to lists in order for the code to work (the current example is how regions are held in and friends). This should just be a matter of sitting down and going through the code (again, trying to be consistent).
o I've tried to avoid 'Lispifying' the code to date (it still looks very similar to the Dylan code). I think I've gone about as far as I can without doing at least a little 'Lispification' (particularly wrt slot accessors), so some of that needs doing too.
o I need to document more, especially if the code starts to diverge from the Dylan implementation since the FunDev documentation will not be so useful anymore.

I think this needs prioritising in the following order: list/vector changes, unit test changes, object construction, lispification with documentation being a common thread over all tasks.

More updates soon (I'm not going to promise since I'm so bad at providing them, but I will promise to try harder :-)

Time flies (reason for lack of updates)

2006-04-25T10:29:00.000Z

Well, 'real life' kind of caught up with me for a while after the new year. I haven't died (yet) or been run over by a bus or anything and I'm back looking at the DUIM port (I was intending to take a break for a week, and here we are four months later...).

I think the break may have been useful, I see some holes in the code that weren't apparent to me before (but then that might be an excuse to concentrate on the interesting side of understanding the windowing model better etc., rather than to actually, you know, get it working).

I'll try to make a set time for posting here in future (maybe Friday afternoons each week) to provide regular progress reports if for no other reason than to highlight any lack of progress to myself so I'll be spurred on to work harder :-)

The Devil's in the Details...

2006-01-07T14:34:00.000Z

Things are starting to come together wrt DUIM. I've started to work on a back end (using LTK at the underlying toolkit) and converted over the tic-tac-toe example program from the Dylan.

No luck so far getting things to run; I've made too many arbitrary and inconsistent conversions throughout the code (mainly regarding when to use lists and when to use vectors); the Dylan DUIM code has a bunch of slots or variables bound to (for example) #(). To a Lisp programmer this looks like a vector, so these fields have been made vectors in the Lisp. Often the usage of the object demands a list however, so I'm getting a few incompatible type exceptions being thrown. The opposite situation is also a problem.

It's just a matter of time until these are all sorted out though (and hopefully not a huge amount of time either).

Also there is a distinct lack of documentation regarding what needs to be implemented for the back end. Until I sit down and go through it properly, and document (which would probably be quicker, but not as much fun as hacking things until they work) there's a good deal of trial and error. Knowing something about McCLIM back ends simplifies things somewhat, but not as much as I'd like!

The immediate TODO is currently:

1. Get the core, back end and tic-tac-toe code to a point where the tic-tac-toe demo works
2. Convert the code to a Lispier style (remove DEFINE-DYLAN-CLASS et al)
3. More unit tests
4. Convert the rest of the example programs, fix anything necessary to get them running
5. Convert the Dylan DUIM tests
6. Examples, documentation, applications
7. More back ends (I'm keen to bash out a "better" Beagle -- perhaps this will lead to improvements in the McCLIM Beagle too)

Somewhere during that list I'll make a release -- hopefully this will be "soon" but I don't feel the code is there just yet. Releasing now would be confusing I think.

I am pretty happy with the state of things as they stand, but will be happier when everything's moved on a way.

So forward with the debugging!

December - January DUIM tasks

2005-12-16T19:50:00.000Z

Spurred on a little by the activity on the gardener's list (see previous post for link) I've decided to juggle my plan of work with regards the DUIM port a little. I had planned to get a decent amount of unit test coverage in before starting work on any back ends; now I think I will follow a different schedule:-

1. Transliterate the vanilla port code and fix up the existing unit tests
2. Write a simple back end (e.g. not necessarily fully functional) -- chances are I'll use Peter Herth's LTK library for this
3. Write unit / regression tests for anything that's obviously broken during writing the back end
4. Put together a 'first release' (maybe register the project on common-lisp.net)
5. Maybe write a simple UI on top of RT (there's one already for McCLIM that I might modify, or a new one) and continue with unit test coverage.
6. Rewrite the API to be more Lisp-like (copying the CLIM API to a great extent for this)
7. Put together an advertised release
8. Eat my own dog food ;-)

Useful DUIM links

2005-12-15T11:43:00.000Z

I mentioned on the cl-gardeners list that I'd provide a couple of DUIM links here today (there's quite an active discussion going on there regarding GUIs at the moment) so here they are:

Functional Object's DUIM Library Reference
Functional Object's DUIM Tutorial
Functional Object's Dylan Extensions Reference (pretty useful if you want to know what some of the stuff in the DUIM code was intended to do)
cll post from SWM that made me say "why not do just that?". That whole thread is pretty interesting I think...
cll post from SWM regarding the problems wrt implementing native look and feel in CLIM.
cll post from SWM hypothesising about UI design in general.
Gwydion Dylan homepage; the original DUIM code is in this repository, along with the Functional Object's Dylan IDE.

The FunO code is released under the LGPL which is handy.

DUIM code conversion 'complete'

2005-12-10T16:39:00.000Z

Subtitle: completeness is in the eye of the beholder :-)

I'm pretty much done converting over the Dylan code to Common Lisp. I've revisited the macros and think that pretty much all of the core code is converted (with one exeption). I've decided to use RT as the testing framework.

The next steps are to write a bunch of tests, and to transliterate what's left of the code into Lisp (all that remains of the core DUIM code is multi-processing related).

Then it's on to providing a back end, writing documentation and putting together an alpha release. I don't like to make timescale estimates, but I'm hoping I will be able to start on a back end in earnest by the end of January. We shall see...

LispOS revisited...

2005-11-29T16:19:00.000Z

I'm not sure quite why, but today I was hit by a wave of nostalgia which caused me to reread some of the posts of the LispOS mailing list (archived at: tunes.org). These mailing archives are responsible for leading me to CLIM, McCLIM then DUIM (in that order) in search of a decent libre Lisp environment.

I had a few thoughts (very few of which are actually original) on what I'd like to see in a Lisp environment (if not a Lisp OS). I'd like to record them for purely personal reasons - so I'm going to.

1. Initially it would have to be based on top of an existing OS (for driver support mainly, and also so that existing (non-Lisp) apps are available). Theoretically this could be any OS (Linux, *BSD, NT, ...) as long as the OS interfaces are sufficiently abstracted.

2. The Lisp world and the native world need to interoperate nicely. For a unix interface this means basically 3 things...

2.1: The Lisp world needs to be able to make use of the available libraries on the host system (at a level of abstraction higher than FFI; lots of library wrapping).
2.2: The native processes need to be able to invoke functionality in the Lisp world direct from the shell (e.g. '> wc -l myfile.txt | some-lisp-fn > output.txt') which would require either a non-standard shell, or a hacked Bash (or whatever). Ideally from the user's point of view it shouldn't be possible to tell which commands are Lisp, and which are native. I suspect this will be easiest in a new shell, but that's unlikely to provide the potential for wide-spread usage I'd like to see.
2.3: The native side needs to be able to call into the Lisp world, as if the Lisp world were just a big shared library.

3. The chances of this kind of environment catching on in the large, in the real world, is pretty small (since most organisations have so much invested already in other technologies such as C and Java (or .net)). So I think it would need to be possible to write code in Lisp, and package that code as Java .class files or C libraries.
This needs to work both ways (i.e. be able to use .class files in the Lisp world, the C interface is already mentioned above). Mechanisms already exist to enable this interoperation but again, further abstraction would be nice.

4. Much of the initial work would appear to be writing FFI glue code or file-processing code so that tools can be developed at particular levels in the architecture (e.g. write a mail client in Lisp, but keep the underlying mail sending / receiving / queuing functionality native, or write a tool to edit passwd files that operates on abstract data (lists? ;-) but that ultimately updates the file itself).

5. It might be necessary to be able to ship the Lisp runtime (sans compiler etc.) as a shared library so software developed in the environment can be used seamlessly by users of the native environment without them having to install a full Lisp development system. (I'm not sure this is actually needed, but in terms of marketing its probably something that would be good, politically.) This would reduce the burden of supporting users in any case (i.e. they wouldn't be able to stuff things up) and Lisp people will have access to the full environment anyhow.
Certainly for small throw-away utilities (perhaps like the passwd editing tool briefly mentioned earlier) its imaginable that non-Lispers won't want a full environment (many Linux users never compile from source for example).

6. Suitable abstractions for accessing hardware devices would be nice so the devices can be accessed from Lisp (apparently directly) but the environment would still use the native drivers. DUIM is obviously the abstraction for the windowing system, but why not abstractions (better than currently available) for accessing all sorts of different devices (mouse, screen, ports, modem, graphics tablet....)?
Obviously this won't be necessary unless people want to take direct control of things at the device level.

7. Perhaps there's a way to make FASLs (more?) self-contained (i.e. instead of building FASLs based on a particular source file, build 1 FASL per package instead (or some other compilation unit)) so that Lisp 'apps' can be loaded on demand (when thinking of many OS processes talking to a single Lisp world, each conceptual 'OS application' implemented in Lisp will need to be loaded into the world) since its unlikely anybody will want to load all available 'Lisp apps' into the core at the same time. Following this logic through it seems like it might be useful to be able to *unload* the FASLs too, perhaps at GC time (for example, the Lisp version of 'wc' might be loaded at some point when the user invokes it from the shell, it probably doesn't need to be kept in core forever afterwards, especially if the user doesn't invoke it very often).

I think the above could provide a solid and useful hosted Lisp environment; once that's available, and the environment is sufficiently developed (i.e. good editor, debugger, stepper, introspection capabilities...) it should be possible to remove things from other layers of the application / OS stack - eventually leading to a full LispOS, possibly.

All that kind of fleshes out my TODO list somewhat:-

. Finish DUIM port to Lisp (I guess at least 2, maybe 3, months)
. Port DUIM extensions (presentations and the other FunO stuff)
. Port DEUCE
. Write Lisp mode for DEUCE
. Write Lisp Listener
. Implement some method for running native apps and pulling any output back into the Listener
. Decent debugger / inspector etc.
. Stuff from the above list

That should keep me going for a few years... ;-)

DUIM port progress

2005-11-23T14:21:00.000Z

Time is flying by. I find it hard to believe that nearly a month has passed since my last post. I think a progress report is in order...

I've tidied the code up significantly from where it was a month ago; the consistency of the code is much improved.

I removed the DEFINE-ABSTRACT-CLASS and added DEFINE-DYLAN-CLASS instead. The latter accepts a list of modifiers (for example, (:sealed :abstract)). These modifiers don't actually do anything but it does mean the Lisp code reads more like the original Dylan.

I've added the recording module to the code base, and moved to an ASDF build which makes life a little easier.

Most of the macros are complete now (although they need a serious amount of testing which should be possible Real Soon Now(tm) (I've created the user and core modules too which should provide a convenient testing sandbox)). Only the really hard (or scary-looking, at least) macros are outstanding and I have a 90% clean compile.

Things left to do before this phase is finalised:-

1. Finish converting the hairy macros (frame definer, command table definer, etc.)
2. Clean the compile up
3. Decide on a test framework (I'm leaning towards Franz' open source testing framework at the moment, but want to look at the Dylan Testworks code too which is in the fundev source tree).
4. Write unit tests and documentation strings for the DUIM functions.
5. Port the DUIM tests from Dylan into Common Lisp and get them running too.

Once that's all done I'm back to my original plan of implementing a back end and starting on application development.

CL-DUIM Progress

2005-10-29T12:12:00.000Z

According to the library.dylan file in the 'core' directory, the following packages need to be defined (forming, I assume, the core of the DUIM functionality):

duim-utilities
duim-geometry
duim-DCs
duim-sheets
duim-graphics
duim-extended-geometry
duim-layouts
duim-gadgets
duim-frames
duim-recording

I've now transliterated all these bar one (duim-recording) into Lisp - of a form. I need to revisit what I've done and standardise on some of the approaches I've taken (for example, should the built in Lisp list and array types be used, or should new Dylan-like objects be built?) because currently some things are rather haphazard (another example - I started converting Dylan's abstract classes into Lisp standard classes (using defclass) and later decided to provide a 'define-abstract-class' macro) so half the code is one way and the other half the other way.

The notable thing missing currently is any back-end code; once what I have has been kicked into shape a little more I'm going to crack on and get something working with Cocoa (hopefully). Then I need to go back and add in the other things I'm missing; recording, presentations, graph, menu and table formatting, support for images, and the 'panes' support (the latter possibly needing to be expedited, I'm not sure yet).

I also need to revisit my automated tests which are already suffering some bit-rot.

Still a long way to go, I'm going to be busy!

My current plan of work consists of:

1) Tidy what's already converted
2) Convert vanilla-backend, and write a concrete (Cocoa) back end
3) Convert the 'extras'
4) Automated tests
5) Port DEUCE to CL (on top of DUIM, obviously)
6) Document, document, document (probably need to write some kind of document authoring tool, or just use something already available. It would be nice to be able to integrate the documentation into some kind of DUIM-based workspace though)

SAD

2005-10-23T01:44:00.000Z

I've just finished the structured analysis and design course my employer sent me on. It was based on UML 2.0 of course and actually wasn't too bad. The instructor (John Conolly) seemed to actually be competent, which was a nice surprise.
My only worry is that this kind of activity won't actually resolve any of the issues we tend to have to deal with when developing software in the commercial world, namely that 'the business' never knows what it actually wants (and even more rarely what it needs) and the developers are never given enough time to write even half-decent code (in our case in Java). Whilst structured analysis / design should help I just can't see us getting the time to do the job properly unfortunately.

UML doesn't look like it will fit in too well with Lisp, but I was expecting that. I suspect with a few tweaks (support for multimethods being the most obvious) UML could be used to represent a Lisp system. Unfortunately dabbling in structured design will pretty much destroy one of the biggest strengths of Lisp; namely the dynamic nature of development and the ability to change running code on the fly.

I think it shouldn't be too hard to dump the structure of a Lisp program into UML though (well - once the limitations of UML wrt Lisp are addressed) so in terms of 'model driven architecture' Lisp and UML should be a pretty good match.

The 3-week course has halted the work I was doing porting DUIM into Lisp. I'll get back to that over the next couple of days. Most of the Dylan code has been converted now, but I need to finish off the macros (Dylan's macro syntax feels very kludgy compared to Lisp's) but that shouldn't take too long. There are a few compatability methods I want to revisit and I'm going to wrap Lisp sequences to make them look more like their Dylan counterparts.
I've also got quite a big job to do documenting the thing (Fundev's documentation is good but I want more detail; think CLIM 2 spec rather than DUIM API reference) and writing some automated tests.