Skip to content

Auto-reloading code with Clojure

Today, I found an awesome little library called ns-tracker, put together by James Reeves. Most of the code was originally written by Stuart Sierra for his Lazytest tool.

ns-tracker provides a function to call for checking if any of your source code has changed and then provides you with the namespaces and their dependencies so that you can plug it straight into clojure’s namespace loading function. It’s brilliant and absolutely changes the way you code, getting a lot closer to Brett Victor’s Inventing on Principle - a worthwhile watch if you haven’t seen it.

Basic usage

(def tracker
  (ns-tracker ["src" "test"]))

(doseq [ns-sym (tracker)]
  (require ns-sym :reload))

This is the quick way of doing a check for code changes - build a tracker with your source code folders and then use it to find modified files and reloading them.

Background tracking

I did some work to make it into robust little worker thread, since I generally want this to keep going even if the mountains fall over (exception made for SIGTERM):

(defn check-namespace-changes [track]
 (try
   (doseq [ns-sym (track)]
     (info "Reloading namespace:" ns-sym)
     (require ns-sym :reload))
   (catch Throwable e (.printStackTrace e)))
   (Thread/sleep 500))

(defn start-nstracker []
 (let [track (tracker/ns-tracker ["src" "checkouts"])]
   (doto
     (Thread.
       #(while true
         (check-namespace-changes track)))
     (.setDaemon true)
     (.start))))

Now I just run (start-nstracker) when my app starts, and I’m done. My work cycle is now edit code, save, see the result, rinse, repeat. I’m using slightly tweaked versions of the code above for the different apps I’m using.

Note here that I have pulled the actual work into a function instead of glomming the lot together – This is so that if I want to tweak the reloading function, I can do that on the fly too.

Lein checkouts for libraries

You’ll also notice that I’m using “checkouts” instead of “tests” – I can quickly make changes to supporting library code and it’ll load that code up too. Leiningen supports having library sources under the checkouts folder, then it will use that instead of the maven dependency. I use symbolic links in there to le me use one central folder per library.

I have 3 apps using some shared library code, and it’s just magical when I save a file all 3 apps instantly have the code loaded.

Swing

Because the changes are so immediate, I just had to put together something I can use to give me a quicker feedback time on UI development:

(defn test-panel [panel & [title]]
 (-> (sw/frame :title (or title "Test Panel") :on-close :dispose :content panel) sw/pack! sw/show!))

(defonce scaffold-frame (atom nil))

(defn scaffold
 "Show or refresh a test panel using the given panel function to build the contents."
 [panel-fn]
 (if (nil? @scaffold-frame)
   (reset! scaffold-frame (test-panel (panel-fn)))
   (sw/config! @scaffold-frame :content (panel-fn))))

This is using seesaw to abstract swing out a bit, but I can include a call to scaffold somewhere at the root of a namespace and on reload, it updates my frame with the new panel, without moving or resizing it anywhere. Brilliant.

Summary

I’m really enjoying this new way of development – makes me feel way more in touch with what I’m building, gets my startup time quicker and the focus in the right place. This lightweight environment also encourages experimentation which gives you the freedom to try new ideas and different ways of doing things to see how they feel, instead of painstakingly trying to guess from a distance.

A look at Clojure Concurrency primitives: delay, future and promise

As I was reading through the excellent book, Clojure Programming book by Chas Emerick, Brian Carper and Christophe Grand, I got to the part on Concurrency and felt that I should write about these primitives as a taste of what concurrency is like in Clojure. I highly recommend reading the book as it explains these concepts far better than I could in this short post!

So, quick rundown of these handy constructs:

Delay

delay wraps up an arbitrary body of code for evaluation at a later stage (when it is required) so that the code in question isn’t run unless asked. This will also cache the result value of the code to prevent another execution. This means that the code is guaranteed to only run once, even if dereferenced concurrently. A small REPL example (the “user>” bit being the REPL prompt) :

user> (def d (delay (println "Hello world!") 42))
#'user/d
user> d
#<Delay@320dafa2: :pending>
user> (realized? d)
false
user> @d
Hello world!
42
user> @d
42
user> (realized? d)
true
user> d
#<Delay@320dafa2: 42>

Here first assign the delay to a var, arbitrarily called ‘d’, if we look at it, we can see it starts in a pending state. We can then check if it has been run with the realized? function – dereferencing it (with the @) causes the code to run and Hello world! printed. Notice that the second @d does not print Hello world! again.

Future

This is a simple way of spinning off a new thread to do some computation or I/O that you will need access to in the (surprise!) future. The call is entirely compatible with delay, so you can swap them out if you want to start the work immediately. If you dereference a future, it will block until the value is available – REPL example :

user> (def do-something (future (Thread/sleep 10000) 28))
#'user/do-something
user> do-something
#<core$future_call$reify__6110@14c55ea: :pending>
user> (realized? do-something)
false

… 10 seconds pass ….

user> (realized? do-something)
true
user> @do-something
28
user> do-something
#<core$future_call$reify__6110@14c55ea: 28>

You can see that realized? is once again useful for when you don’t want to incur the blocking penalty and that you didn’t need to dereference do-something for the value to become realized.

Promise

Promises are used in a similar way to delay or future in that you dereference them for a value, can check if they have a value with realized? and they block when you dereference them if they don’t have a value until they do. Where they differ is that you don’t immediately give them a value, but provide them with one by calling deliver:

user> (def p (promise))
#'user/p
user> (realized? p)
false
user> (deliver p "as-promised")
#<core$promise$reify__6153@66650e56: "as-promised">
user> (realized? p)
true
user> @p
"as-promised"

Of course, calling @p will block the REPL until you open a new one and deliver it (provided you’re using swank or something that can have multiple open REPL threads). There are some really neat ways to combine the promise with future to get nice and declarative concurrency semantics. But I’ll leave that as an exercise for you.

I really enjoy how the dereferencing work exactly the same for future, delay and promise as they do for the Clojure atom and ref types.

Clojure on Bukkit minecraft server

 

This morning, I finally managed to get the basic skeleton of the bukkit clojure plugin “host” working correctly. Clojure doesn’t like having to straddle across classloaders in order to find the resources it needs! I can’t say I blame it :)

It’s a fantastic feeling being able to code plugin stuff on the fly – and as a glimpse into that world, I’ve created a small timelapse video demo’ing some basic possibilities. Apply some imagination though, I’m sure you’ll come up with some awesome stuff..

Most importantly for me is the exploratory nature of Clojure – It fits very well into the nature of Minecraft. It’s thrilling to be able to discover and tweak code for your plugin on the fly.. But see for yourself: Make it full screen btw.

And yes. I get killed by a spider.

Check out the code that makes this little thing tick:

The Clojure plugin source is at : https://github.com/CmdrDats/clj-minecraft

The code (memorystone v2) used in this video: https://github.com/CmdrDats/clj-memorystone

I’m planning to build a leiningen template so that starting a new clojure bukkit plugin is as simple as lein new bukkit-plugin [plugin-name]. I’m also re-implementing my MemoryStone plugin just so that I have a base to work from in order to really flesh out the shared clojure library.

One idea I’m toying with is making a more intuitive way of registering listeners to Bukkit. I’m not sure if I’ll actually end up with anything other than a convenience wrapper function for .registerEvent, but it’s worth messing with.

As you can see in the clj-minecraft project, I’m wanting to provide a fairly solid set of basic plugin building blocks so that you have simple access to the fundemental building blocks of plugins. My goal is also to implement and wrap some common plugins, such as SpoutCraft and the various Economy managers.

I would like the 3rd party libraries to be supported in a way that automatically “switches off” when the plugins aren’t present, so that plugin developers don’t have to worry about it besides actually making sure they have alternative methods of using their plugins.. But we’ll see as I thresh this out.

Feedback will be hugely appreciated and I always welcome a helping hand! So please, contact me, fork my repo’s and make pull requests!

 

Clojure TopCoder Timelapse, Rainy Road, SRM 525, Div II, Level 1

I figured out how to make a timelapse video. Yay! Then I decided to tackle another topcoder challenge. RainyRoad -

At the beginning of the video, you’ll notice that I have my browser open – I hadn’t read the problem statement yet, which is dangerous since it could be a really tough problem!

Turns out, it was a dead simple problem. At first it looked like I needed to do some pathfinding (was about to brush up on a*).. but then it was just a matter of checking if both sides of the road is occupied with a “W” or not.. Fairly simple.

So I recorded myself building two solutions and have checked in another (better, imho) solution into the mix. The video is recorded at 5 fps, and each frame is 5 seconds apart, so each second is 25 real seconds..

Take note that I had not prepared at all for the problem. I hadn’t created a project yet or setup anything on my emacs. In the first second or two you’ll see me ‘lein new’ and cd to the new project folder. And then a little later start the swank server for emacs to connect to (about 15s in).

My first approach was to map pairs of path together so that I could just test each pair individually – the second approach, i didn’t bother with that, just doing a for over and index range, concating the letters together and checking for “WW”, then filtering the result for the word “NO” and check the count

The third approach, which you can view here as the function (another-reachable) was a lot more interesting – It interleaved the characters, then used a loop / recur to take pairs off the resulting array, concat the characters and check for “WW” – returning immediately with “NO” if not possible to cross.

You’ll notice that I’m running functions in a *slime-repl clojure* buffer – that’s connected to my running project (lein swank), so I could fire off a thread and have it continue without me being connected.. that’s pretty cool :)

Otherwise – enjoy the video – As always, feedback is really appreciated :)

Clojure + TopCoder SRM

I’ve recently started delving into the interesting world of Functional Programming. This was sparked by an amazing video by Rich Hickey called “Simple Made Easy” – I highly advise every programmer to listen closely, this man has his head screwed on right and has given a lot of thought to his subject matter. Even if you’re not interested in Clojure, as a developer, I believe every one of his talks are highly valuable and worthwhile pondering.

In line with adopting Clojure, I’ve mostly switched to Emacs. I still use PhpStorm for PHP development and Eclipse for Java development – I would prefer to move away from them, but emacs has a rather significant “bootstrap” time, so I don’t think I’d really make much of a return on investment there.

Rather, I’m leveraging the excellent emacs clojure-mode with paredit, slime, lein and swank which really makes clojure development shine. I still can’t get over how amazing on the fly development is with emacs + clojure. I have to remind myself that I’m actually working on a program every now and then and not just having fun.

On that vein, my biggest challenge now is wrapping my head around Functional Programming thinking. And I don’t mean just how to do recursions, but to really get to grips with how to leverage the powerful constructs of Clojure – lazy evaluation, lazy sequences, macro’s, immutability, lambda functions, agents, refs, atoms and so on. There’s a lot of good stuff locked away just behind the doors of FP if I just care to take the time to pick the lock :)

To that end, I’ve decided to start doing past TopCoder SRM challenges in Clojure, as thinking practice. I could be doing something, you know, actually useful. But I’m looking for something that’s highly mentally stimulating but small and of a throw-away nature.

I’ve decided to host my solutions at GitHub so that others can see my code, maybe comment and perhaps even learn from the code.. Or at least it’s just there :P

Read more…

Hosting Change

I’ve finally managed (after almost 2 years) to make some time and get my dats.co.za domain updated. I was using HostRocket, but the 2 year renewal is coming up soon and I felt I didn’t use the facilities enough to warrant the price tag any longer.

In order to keep the functionality of my dats.co.za domain though, I needed these things :

  • DNS Server
  • A new blog server
  • An app server of some sort (so I can host code)
  • Basic site

The DNS server is by far the toughest of the lot, lucky I own the dats.co.za domain directly with the co.za registrar, so I could make changes there without much hassle. I looked at, and almost switched to, http://freedns.afraid.org/, but then I found out that they share the domain amongst all their users. Which is quite terrible.

I eventually settled on http://www.zoneedit.com, which is free up to a certain amount of traffic. I’m happy with that since I don’t think my dats.co.za traffic is that great. ZoneEdit is quite minimalistic, but it works extremely well, so I’m very chuffed.

Another thing about ZoneEdit that I mustn’t forget to mention is that they support WebForwards and EmailForwards directly – which is great. I don’t know if all dns servers provide this, but it works like a charm!

Next up, a blog server. A major issue with running and hosting my own blog online is that hackers have an absolute field day – you need to keep on top of things and keep your software up to date so that they don’t exploit all sorts of arbitrary vulnerabilites. I found two or three folders on my dats.co.za site that were using my site as part of a botnet. fantastic. Not to mention the times I’ve had to entirely restore my site from backup because some script kiddie thought they were clever.

Anyway, to my point – I’m enjoying using an externally provided blogging solution, since the admin I have to do on my side is minimal.

It was a tough choice between wordpress and blogger – WordPress has more advanced features, and I’m familiar with it – where blogger is simpler and is a Google product, so integrates more easily and tightly with other Google products. In the end, I customized wordpress and blogger to what I thought was the nicest way, and WordPress just won me over. So here I am :) (To be honest,  I think it was the fact that I could nicely display my buzzes on the side – blogger didn’t do as good a job on that – ironic, no?)

App server was a simple choice – Google App Engine – It supports Java now, so I’m going to be having a field day!

Basic site – I settled on using Google Sites since it already integrates well with the Google App Engine and I could see that it handles nicely when you point your DNS server to it. I enjoyed setting up the site, and while there’s more that could be done there, it serves it’s purpose well. Minimal effort for a well functioning site, what more could I ask for.

On the flip side I don’t have my Eashi wiki any longer, which is a little sad, but it was hardly being used so I think I can get away with running that locally, should I ever want to. I also don’t have the 5gb space any longer, but I don’t think that will ever be a problem, with the kind of free space floating around the internet.

Bottom line, my single point server from HostRocket has been fragmented across an array of free online services. That fragmenting is great because I get to drop and switch if I need to, without paying a cent.

Now, I just need to figure out how to do some kind of automated backup on all of this :P oh well, if all my stuff gets wiped from the face of the earth I doubt anyone will even really notice.

Legoduino!

Tonight I got a bee in my bonnet again.. was curious to see how the arduino would fit on a lego plate (well, technically not quite reaal lego, but let’s pretend for a while), and how I’d go about breaking that out into usable lego chunks..

Again, rather than bore you to tears with the explicit details, I’ll let the pictures do (most of) the talking :

Arduino Duemilanove, mounted on a lego plate.

As you can see, there’s an extra mounting hole – I had started off mounting it to the bottom of the plate, but I think that was just a disaster – there’s not enough space on the plate to break out properly with my paperclip method and trying to hold the whole thing up with just the top studs (against gravity) is precarious at best.. So I decided to put it on top.

Here’s the bottom view, to show the mountings :

I then decided I’ll use some header pins and break that out into sections – Luckily I have a bunch of old serial ports with their connector cables that I could salvage :D (thanks to the guys at work donating me a bunch of old pc parts and stuff) :

If you’re sharp, you might notice the twisted red wire and the odd red wire in the next .. and if you’re an electronic engineer, you’ll probably also instantly guess that I’ve chosen the red wires on each set of connectors to go to ground.

Very handily (is that even a word?), there are 3 ground pins on the arduino, so perfect, since I’ve got 3 connectors out :D

Here’s the finished product :

Now I just have to figure out how to connect those ends up to lego and break it in from there…. Any ideas?

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: