Travelling to Japan is a long and very tedious journey. I have a habit of packing lots of things in my hand luggage to keep me entertained and then only using one of them. As was the case this time when I spent the entire journey reading on my Kindle “Why Buddhism is True: The Science and Philosophy of Meditation and Enlightenment” by Robert Wright.

It was a fascinating read and I intend to get the audiobook and listen to it again. In many ways it set the tone for my visit and attempt to try and get further with meditation as I had attempted before. As is usual I did not get as far as I wanted although maybe I got further than I thought I would.

I read (or started to read) a few other books on mediation whilst out there but none of them grasped me as much as Robert Wright’s book – maybe as his focuses a lot on the theory and not so much the practice. Still, I still probably learnt more than I thought I would.

Finally I discovered Alan Watts who died in 1973 aged 58 (just seven years older than I am now). I’m sure I’ve heard him before and I discovered him by accident whilst looking for something else completely different. He seemed to be a complex and engaging character with an interesting background and life. I’ve started listening to some recordings of his lectures. He’s very odd and interestingly described himself as a “philosophical entertainer”. I don’t understand all he says and believe much less but he is very thought provoking.

I am going to start using this blog to start writing more about what I am learning and thinking, happy in the knowledge that it’s public but unlikely to be read by anybody – certainly anybody who knows me.

Emacs Prelude with Cygwin

Some notes on installing Emacs prelude with Cygwin. To quote the prelude website at http://batsov.com/prelude/

“Emacs Prelude has the goal to ease the initial Emacs setup process and to provide you with a much more powerful and productive experience than the one you get out of the box. “

Although I’ve been using Emacs for nearly fourteen years now I decided to ditch the masses of cruft that had built up over the years and launch into a new setup. Prelude was a nice starting point for me, giving me a nice working environment into which I was able to merge in some of the cruft that I realised I couldn’t live without. Many thanks Bozhidar.

I’m using Cygwin 1.17.7-1 for the record. I am going to assume you can follow the cygwin install instructions and the prelude install instructions. If you do get stuck feel free to ask. I’m no Cygwin expert (or emacs expect) so don’t expect too much ;). This post just lists some stuff you might run into under windows.

First you’re going to need to install Cygwin (obviously) and the following extra packages:

  • emacs-w32
  • curl
  • git
The entire prelude install process runs through a downloaded shell script. curl will give you en error “(77) error setting certificate verify locations”. You can switch this off by doing:
   echo insecure >> ~/.curlrc
Obviously that’s not the right way of doing it. Feel free to let me know the proper way. Similarly you need to tell git to be wild and free about what it’s connecting to. Once again this is NOT the right way to do this but for the record
   git config --system http.sslverify false

The next issue that might trip you up is if your home directory has spaces in it.  My home directory is defined as

   /home/Ian J Cottee/

and that will make the installer upset. In this case I felt the best thing was to fix the problem for good as no doubt other utils outside of prelude might be thrown by this.

   juice: / $ cd /home
   juice: /home $ ls
   Ian J Cottee/
   juice: /home $ ln -s Ian\ J\ Cottee/ ian

I then modified /etc/passwd so the home directory was set to /home/ian

If you now start emacs you should see prelude setting itself up. It’s possible you may need to remove your existing .emacs.d (in my case it got created during one of the failed processes). If you’ve been using emacs before under Cygwin you’ll want to back it up.

The final issue is you may see in the console messages such as:

   0 [main] emacs-w32 9732 child_info_fork::abort:  

and plenty more gumph and in emacs messages such as:

   File error: Doing vfork, resource temporarily unavailable

To fix this follow the instructions at http://cygwin.wikia.com/wiki/Rebaseall

And there you go. You should be in business. Note I’ve only just got it running here. I’ll try and keep this post updated with all the pitfalls I will no doubt run into.







The joy of legacy Zope/ZODB systems

I have a large number of legacy systems that – when all other avenues fail – become my responsibility to sort out. Some of those are very old Zope systems written by others and which never fail to reduce me to tears. This morning I came across some particularly good design decisions which I thought I’d share. Yes, that ‘good’ is sarcasm.

First of all let’s remember that Zope, by default, uses the ZODB. In my past lives when I used to use Zope I used to use it as a frontend to Postgres (which sounds nuts now but at the time we didn’t have lots of fancy MVC frameworks to spend our hours with). The legacy systems I have currently been gifted with however use the ZODB to save their data. You could argue that for content management systems (which is what we’re talking about here) that the ZODB is not a bad fit. I’m not going to argue one way or the other, although my own personal point of view is that it’s shit) – however, for the purposes of today’s “issue” the data was real data – i.e. information that would sit quite comfortably in a relational database as god intended.

We’re talking about parent data with child records. This particular issue was that the customer had made a mistake and I needed to remove a bunch of child records from one particular parent record and this had to be done by the sixth of March or the world would end or something similar. Fair enough, so I poke around the code and the database.

Poking around a ZODB database isn’t like poking around a relational database. You can’t look at a database schema and say “these records in this table have these fields”. All you can say is that this record in this big bucket of crap has these fields. And the record next to it could be the same type of record with different fields. Or a completely different type of record. So you poke around and try to work out what is held where. If you fancy a laugh you take a look at the code that writes the records and try and make some sense of it but frankly you’re better off pulling the nerve endings away from your fingers one by one, with a rusty pair of pliers.

So eventually I find the child records and I find a method which is called something like ‘deleteChildRecord’. It turns out this doesn’t just delete child records – it deletes parent records as well because in this brave ZODB world they’re all sitting in the bucket of slop together. Which is OK in the scheme of things because by this time its only 05:30 in the morning and we’re beyond caring. By the way – there is a doc string on deleteChildRecord but it doesn’t seem to make much sense … at all. Then I realise the same doc string is used to document virtually every method in the file. Somebody copied and pasted the same doc string twenty times and never thought it useful to change it to something else. But that’s OK. We’re used to that.

So I write some code that works out which child records to delete and run it and get the client to check. They respond saying that when they look at the parent record they can still see ‘stubs’ of the deleted child records. The codes of the deleted records are still showing against the parent but no details of the child records. This doesn’t surprise me that much because in my ZODB world it’s very hard to get rid of data. Like the brown stains around the porcelain after a particularly heavy dump, there’s always some lingering remnant that remains in the crusty crevices of the database even after repeated flushings. Often it’s because the indexing system that Zope and it’s content management framework (CMF) uses hasn’t removed it’s indexes of metadata even when the real record has been removed. Because you should never query the database directly (unless you fancy wallowing through 50GB of binary data record by record) code will just query the indexes (known as the catalog) which contains the main info you need and will then pull out the records for each catalog item it finds.

Am I boring you? By the way, it just occurred to me that the best way to ensure that all traces of a particular record are removed from Zope are to tell it that’s it’s vitally important that it should be kept. I can guarantee you’ll never see it again. Anyway – back to the story. Oh god. So I think … the catalog has not updated itself. I will rebuild the catalog. This I do. This is a simple thing to do – you just click on a button labelled ‘Rebuild catalog’.

I now have no data.

You see, whatever genius designed this part of the system decided to store the data in the place where you’re supposed to just store the indexes and metadata of a record. Just as you thought you were winning and the baddie had been despatched to the lower reaches of hell you see a shape in the window and it turns out you’re still thirty minutes from the final credits and you’re not halfway through the body count yet.

Fortunately, like the seasoned campaigner I am, I am not doing this on the live system – having transferred the 50GB bag off poo to a staging site before commencing this exercise. Hell, I don’t log onto a bloody legacy Zope site of ours without taking a backup, dumping it to tape and moving it 100 miles offsite.

Right start again and lets start looking at the code.

I can see code that creates parent records. I can see code that creates child records (all got the same doc strings). I can actually see quite a lot of code that creates child records. And I can see code that deletes child records (with the same doc string). Unfortunately that code doesn’t tell a parent record that the child record has been removed so if you do actually use that routine you’ll find the system fails spectacularly the next time you try and view anything. OK – so … the parent record stores the list of related children as a tuple of id’s. So we write some code that takes the tuple (which is immutable) to a list, and then modifies the list every time we remove a child record, converts it back to a tuple at the end and then writes it back to the parent record when we finish.

But of course that still doesn’t work because things such as ‘child count’ are not calculated automatically – they’re stored as properties as well on the parent record. So we manually count the records and update the child count property but then find that even though we have updated that – we have two other counts which are held as two other fucking properties, one for the two possible type of child records we can have and the parent record is wetting itself in the only way it knows how by vomiting python tracebacks over the screen.

So in short when you delete a child record you have to manually a) tell the parent record the child record has gone b) tell the parent record to decrement a specific count for the type of child record you have deleted and c) tell the parent record to manually decrement a specific count for the total of child records you now have.

This I do – give the results to the client who says it all looks fine apart from one record which they don’t recognise the code for and has a completely different child type assigned to it from anything else and would I know why that was?

No, I don’t. I really don’t. I don’t understand this data, I don’t understand the structure (partly because I don’t think there is one). Frankly I hardly know where the floor is at this point and even such concepts as light and dark have gone hazy.