Skip to content

Auto-reloading code with Clojure

Today, I found an awesome little library called ns-tracker, put together by James Reeves. Most of the code was originally written by Stuart Sierra for his Lazytest tool.

ns-tracker provides a function to call for checking if any of your source code has changed and then provides you with the namespaces and their dependencies so that you can plug it straight into clojure’s namespace loading function. It’s brilliant and absolutely changes the way you code, getting a lot closer to Brett Victor’s Inventing on Principle – a worthwhile watch if you haven’t seen it.

Basic usage

(def tracker
  (ns-tracker ["src" "test"]))

(doseq [ns-sym (tracker)]
  (require ns-sym :reload))

This is the quick way of doing a check for code changes – build a tracker with your source code folders and then use it to find modified files and reloading them.

Background tracking

I did some work to make it into robust little worker thread, since I generally want this to keep going even if the mountains fall over (exception made for SIGTERM):

(defn check-namespace-changes [track]
 (try
   (doseq [ns-sym (track)]
     (info "Reloading namespace:" ns-sym)
     (require ns-sym :reload))
   (catch Throwable e (.printStackTrace e)))
   (Thread/sleep 500))

(defn start-nstracker []
 (let [track (tracker/ns-tracker ["src" "checkouts"])]
   (doto
     (Thread.
       #(while true
         (check-namespace-changes track)))
     (.setDaemon true)
     (.start))))

Now I just run (start-nstracker) when my app starts, and I’m done. My work cycle is now edit code, save, see the result, rinse, repeat. I’m using slightly tweaked versions of the code above for the different apps I’m using.

Note here that I have pulled the actual work into a function instead of glomming the lot together – This is so that if I want to tweak the reloading function, I can do that on the fly too.

Lein checkouts for libraries

You’ll also notice that I’m using “checkouts” instead of “tests” – I can quickly make changes to supporting library code and it’ll load that code up too. Leiningen supports having library sources under the checkouts folder, then it will use that instead of the maven dependency. I use symbolic links in there to le me use one central folder per library.

I have 3 apps using some shared library code, and it’s just magical when I save a file all 3 apps instantly have the code loaded.

Swing

Because the changes are so immediate, I just had to put together something I can use to give me a quicker feedback time on UI development:

(defn test-panel [panel & [title]]
 (-> (sw/frame :title (or title "Test Panel") :on-close :dispose :content panel) sw/pack! sw/show!))

(defonce scaffold-frame (atom nil))

(defn scaffold
 "Show or refresh a test panel using the given panel function to build the contents."
 [panel-fn]
 (if (nil? @scaffold-frame)
   (reset! scaffold-frame (test-panel (panel-fn)))
   (sw/config! @scaffold-frame :content (panel-fn))))

This is using seesaw to abstract swing out a bit, but I can include a call to scaffold somewhere at the root of a namespace and on reload, it updates my frame with the new panel, without moving or resizing it anywhere. Brilliant.

Summary

I’m really enjoying this new way of development – makes me feel way more in touch with what I’m building, gets my startup time quicker and the focus in the right place. This lightweight environment also encourages experimentation which gives you the freedom to try new ideas and different ways of doing things to see how they feel, instead of painstakingly trying to guess from a distance.

A look at Clojure Concurrency primitives: delay, future and promise

As I was reading through the excellent book, Clojure Programming book by Chas Emerick, Brian Carper and Christophe Grand, I got to the part on Concurrency and felt that I should write about these primitives as a taste of what concurrency is like in Clojure. I highly recommend reading the book as it explains these concepts far better than I could in this short post!

So, quick rundown of these handy constructs:

Delay

delay wraps up an arbitrary body of code for evaluation at a later stage (when it is required) so that the code in question isn’t run unless asked. This will also cache the result value of the code to prevent another execution. This means that the code is guaranteed to only run once, even if dereferenced concurrently. A small REPL example (the “user>” bit being the REPL prompt) :

user> (def d (delay (println "Hello world!") 42))
#'user/d
user> d
#<Delay@320dafa2: :pending>
user> (realized? d)
false
user> @d
Hello world!
42
user> @d
42
user> (realized? d)
true
user> d
#<Delay@320dafa2: 42>

Here first assign the delay to a var, arbitrarily called ‘d’, if we look at it, we can see it starts in a pending state. We can then check if it has been run with the realized? function – dereferencing it (with the @) causes the code to run and Hello world! printed. Notice that the second @d does not print Hello world! again.

Future

This is a simple way of spinning off a new thread to do some computation or I/O that you will need access to in the (surprise!) future. The call is entirely compatible with delay, so you can swap them out if you want to start the work immediately. If you dereference a future, it will block until the value is available – REPL example :

user> (def do-something (future (Thread/sleep 10000) 28))
#'user/do-something
user> do-something
#<core$future_call$reify__6110@14c55ea: :pending>
user> (realized? do-something)
false

… 10 seconds pass ….

user> (realized? do-something)
true
user> @do-something
28
user> do-something
#<core$future_call$reify__6110@14c55ea: 28>

You can see that realized? is once again useful for when you don’t want to incur the blocking penalty and that you didn’t need to dereference do-something for the value to become realized.

Promise

Promises are used in a similar way to delay or future in that you dereference them for a value, can check if they have a value with realized? and they block when you dereference them if they don’t have a value until they do. Where they differ is that you don’t immediately give them a value, but provide them with one by calling deliver:

user> (def p (promise))
#'user/p
user> (realized? p)
false
user> (deliver p "as-promised")
#<core$promise$reify__6153@66650e56: "as-promised">
user> (realized? p)
true
user> @p
"as-promised"

Of course, calling @p will block the REPL until you open a new one and deliver it (provided you’re using swank or something that can have multiple open REPL threads). There are some really neat ways to combine the promise with future to get nice and declarative concurrency semantics. But I’ll leave that as an exercise for you.

I really enjoy how the dereferencing work exactly the same for future, delay and promise as they do for the Clojure atom and ref types.

Clojure on Bukkit minecraft server

 

This morning, I finally managed to get the basic skeleton of the bukkit clojure plugin “host” working correctly. Clojure doesn’t like having to straddle across classloaders in order to find the resources it needs! I can’t say I blame it 🙂

It’s a fantastic feeling being able to code plugin stuff on the fly – and as a glimpse into that world, I’ve created a small timelapse video demo’ing some basic possibilities. Apply some imagination though, I’m sure you’ll come up with some awesome stuff..

Most importantly for me is the exploratory nature of Clojure – It fits very well into the nature of Minecraft. It’s thrilling to be able to discover and tweak code for your plugin on the fly.. But see for yourself: Make it full screen btw.

And yes. I get killed by a spider.

Check out the code that makes this little thing tick:

The Clojure plugin source is at : https://github.com/CmdrDats/clj-minecraft

The code (memorystone v2) used in this video: https://github.com/CmdrDats/clj-memorystone

I’m planning to build a leiningen template so that starting a new clojure bukkit plugin is as simple as lein new bukkit-plugin [plugin-name]. I’m also re-implementing my MemoryStone plugin just so that I have a base to work from in order to really flesh out the shared clojure library.

One idea I’m toying with is making a more intuitive way of registering listeners to Bukkit. I’m not sure if I’ll actually end up with anything other than a convenience wrapper function for .registerEvent, but it’s worth messing with.

As you can see in the clj-minecraft project, I’m wanting to provide a fairly solid set of basic plugin building blocks so that you have simple access to the fundemental building blocks of plugins. My goal is also to implement and wrap some common plugins, such as SpoutCraft and the various Economy managers.

I would like the 3rd party libraries to be supported in a way that automatically “switches off” when the plugins aren’t present, so that plugin developers don’t have to worry about it besides actually making sure they have alternative methods of using their plugins.. But we’ll see as I thresh this out.

Feedback will be hugely appreciated and I always welcome a helping hand! So please, contact me, fork my repo’s and make pull requests!

 

Clojure TopCoder Timelapse, Rainy Road, SRM 525, Div II, Level 1

I figured out how to make a timelapse video. Yay! Then I decided to tackle another topcoder challenge. RainyRoad –

At the beginning of the video, you’ll notice that I have my browser open – I hadn’t read the problem statement yet, which is dangerous since it could be a really tough problem!

Turns out, it was a dead simple problem. At first it looked like I needed to do some pathfinding (was about to brush up on a*).. but then it was just a matter of checking if both sides of the road is occupied with a “W” or not.. Fairly simple.

So I recorded myself building two solutions and have checked in another (better, imho) solution into the mix. The video is recorded at 5 fps, and each frame is 5 seconds apart, so each second is 25 real seconds..

Take note that I had not prepared at all for the problem. I hadn’t created a project yet or setup anything on my emacs. In the first second or two you’ll see me ‘lein new’ and cd to the new project folder. And then a little later start the swank server for emacs to connect to (about 15s in).

My first approach was to map pairs of path together so that I could just test each pair individually – the second approach, i didn’t bother with that, just doing a for over and index range, concating the letters together and checking for “WW”, then filtering the result for the word “NO” and check the count

The third approach, which you can view here as the function (another-reachable) was a lot more interesting – It interleaved the characters, then used a loop / recur to take pairs off the resulting array, concat the characters and check for “WW” – returning immediately with “NO” if not possible to cross.

You’ll notice that I’m running functions in a *slime-repl clojure* buffer – that’s connected to my running project (lein swank), so I could fire off a thread and have it continue without me being connected.. that’s pretty cool 🙂

Otherwise – enjoy the video – As always, feedback is really appreciated 🙂

Clojure + TopCoder SRM

I’ve recently started delving into the interesting world of Functional Programming. This was sparked by an amazing video by Rich Hickey called “Simple Made Easy” – I highly advise every programmer to listen closely, this man has his head screwed on right and has given a lot of thought to his subject matter. Even if you’re not interested in Clojure, as a developer, I believe every one of his talks are highly valuable and worthwhile pondering.

In line with adopting Clojure, I’ve mostly switched to Emacs. I still use PhpStorm for PHP development and Eclipse for Java development – I would prefer to move away from them, but emacs has a rather significant “bootstrap” time, so I don’t think I’d really make much of a return on investment there.

Rather, I’m leveraging the excellent emacs clojure-mode with paredit, slime, lein and swank which really makes clojure development shine. I still can’t get over how amazing on the fly development is with emacs + clojure. I have to remind myself that I’m actually working on a program every now and then and not just having fun.

On that vein, my biggest challenge now is wrapping my head around Functional Programming thinking. And I don’t mean just how to do recursions, but to really get to grips with how to leverage the powerful constructs of Clojure – lazy evaluation, lazy sequences, macro’s, immutability, lambda functions, agents, refs, atoms and so on. There’s a lot of good stuff locked away just behind the doors of FP if I just care to take the time to pick the lock 🙂

To that end, I’ve decided to start doing past TopCoder SRM challenges in Clojure, as thinking practice. I could be doing something, you know, actually useful. But I’m looking for something that’s highly mentally stimulating but small and of a throw-away nature.

I’ve decided to host my solutions at GitHub so that others can see my code, maybe comment and perhaps even learn from the code.. Or at least it’s just there 😛

Read more…

Hosting Change

I’ve finally managed (after almost 2 years) to make some time and get my dats.co.za domain updated. I was using HostRocket, but the 2 year renewal is coming up soon and I felt I didn’t use the facilities enough to warrant the price tag any longer.

In order to keep the functionality of my dats.co.za domain though, I needed these things :

  • DNS Server
  • A new blog server
  • An app server of some sort (so I can host code)
  • Basic site

The DNS server is by far the toughest of the lot, lucky I own the dats.co.za domain directly with the co.za registrar, so I could make changes there without much hassle. I looked at, and almost switched to, http://freedns.afraid.org/, but then I found out that they share the domain amongst all their users. Which is quite terrible.

I eventually settled on http://www.zoneedit.com, which is free up to a certain amount of traffic. I’m happy with that since I don’t think my dats.co.za traffic is that great. ZoneEdit is quite minimalistic, but it works extremely well, so I’m very chuffed.

Another thing about ZoneEdit that I mustn’t forget to mention is that they support WebForwards and EmailForwards directly – which is great. I don’t know if all dns servers provide this, but it works like a charm!

Next up, a blog server. A major issue with running and hosting my own blog online is that hackers have an absolute field day – you need to keep on top of things and keep your software up to date so that they don’t exploit all sorts of arbitrary vulnerabilites. I found two or three folders on my dats.co.za site that were using my site as part of a botnet. fantastic. Not to mention the times I’ve had to entirely restore my site from backup because some script kiddie thought they were clever.

Anyway, to my point – I’m enjoying using an externally provided blogging solution, since the admin I have to do on my side is minimal.

It was a tough choice between wordpress and blogger – WordPress has more advanced features, and I’m familiar with it – where blogger is simpler and is a Google product, so integrates more easily and tightly with other Google products. In the end, I customized wordpress and blogger to what I thought was the nicest way, and WordPress just won me over. So here I am 🙂 (To be honest,  I think it was the fact that I could nicely display my buzzes on the side – blogger didn’t do as good a job on that – ironic, no?)

App server was a simple choice – Google App Engine – It supports Java now, so I’m going to be having a field day!

Basic site – I settled on using Google Sites since it already integrates well with the Google App Engine and I could see that it handles nicely when you point your DNS server to it. I enjoyed setting up the site, and while there’s more that could be done there, it serves it’s purpose well. Minimal effort for a well functioning site, what more could I ask for.

On the flip side I don’t have my Eashi wiki any longer, which is a little sad, but it was hardly being used so I think I can get away with running that locally, should I ever want to. I also don’t have the 5gb space any longer, but I don’t think that will ever be a problem, with the kind of free space floating around the internet.

Bottom line, my single point server from HostRocket has been fragmented across an array of free online services. That fragmenting is great because I get to drop and switch if I need to, without paying a cent.

Now, I just need to figure out how to do some kind of automated backup on all of this 😛 oh well, if all my stuff gets wiped from the face of the earth I doubt anyone will even really notice.

Legoduino!

Tonight I got a bee in my bonnet again.. was curious to see how the arduino would fit on a lego plate (well, technically not quite reaal lego, but let’s pretend for a while), and how I’d go about breaking that out into usable lego chunks..

Again, rather than bore you to tears with the explicit details, I’ll let the pictures do (most of) the talking :

Arduino Duemilanove, mounted on a lego plate.

As you can see, there’s an extra mounting hole – I had started off mounting it to the bottom of the plate, but I think that was just a disaster – there’s not enough space on the plate to break out properly with my paperclip method and trying to hold the whole thing up with just the top studs (against gravity) is precarious at best.. So I decided to put it on top.

Here’s the bottom view, to show the mountings :

I then decided I’ll use some header pins and break that out into sections – Luckily I have a bunch of old serial ports with their connector cables that I could salvage 😀 (thanks to the guys at work donating me a bunch of old pc parts and stuff) :

If you’re sharp, you might notice the twisted red wire and the odd red wire in the next .. and if you’re an electronic engineer, you’ll probably also instantly guess that I’ve chosen the red wires on each set of connectors to go to ground.

Very handily (is that even a word?), there are 3 ground pins on the arduino, so perfect, since I’ve got 3 connectors out 😀

Here’s the finished product :

Now I just have to figure out how to connect those ends up to lego and break it in from there…. Any ideas?

Legotronics

Something that’s been intriguing me for a couple of days now is the possibility of integrating classic electronics directly into Lego. A quick prototype kind of thing where you can just clip things in and out and have them work.

Now, I could fork out a sizeable amount of cash and get a Mindstorms NXT 2.0, or, I could use the Lego I already have, my dremel and some paperclips lying around to make things happen..

I’ve got an Arduino, so I would like to break that out directly into Lego. My first obvious question is how to modify a brick with simple materials easily on hand (and a high-speed, teensy tiny drill press) and get them to be quickly stackable to build a circuit out of them.

I’ve seen a solution out there, using small nails and then clipping wires onto them, I’m not sure I like that approach though – way too many fickle wires to worry about..

A couple of minutes at the workbench, fiddling, and here’s what I came up with – Holes on top and on the sides, bend paperclip, and voila. It’s kinda springy too! I’ll let the pictures do the talking:

Top view:

Bottom View:

Two clipped together, has a sturdy electrical connection:

So will see where this ends up!

PS. Yes, I’m mauling my spare MegaBlocks which I consider aliens to my collection first 🙂 No sense in experimenting with the real thing just yet!

Microprocessors.. Part 1

Frans (my youngest brother) has me suddenly hooked on microprocessors, and now I have a whole design in my head.. Thus a project is born.. Not sure what to call it yet, but the name that seems to stick in my head is ‘Raven’, so we’ll see how it pans out?

From the top – My dad gave me a ‘Fundamental Electronics’ handbook so I can get to grips with the basics and not lose my head in a random electrical explosion, and Frans gave me an Atmel AVR STK500 board with an Atmega48 processor to mess with.

My end goal of my project is this : Build a modular rapid prototyping system, where I can slap in components like wifi, flashdisks, gps’es, motors, etc together via a unified interface (USB seems a logical choice) and quickly code something (in java, of course) to make it all work together.

As far as rapid prototyping languages go, assembler is definitely out of the question with C not far behind.. My choice of language is obviously Java, so I have my eyes fixed on a Sun SPOT – The $750 price tag isn’t going to happen plus getting it here is going to prove painful, so another solution is required.

After a bit of research, I found the Squawk VM, which (hey, surprise) is used on the Sun SPOT – perfect for the job. Now I need to find a chip that can support it – the ARM920T is what the SPOT uses, so something based on that design should work with the least amount of hassles.

Googling somewhat finally turned up the Atmel AT91RM9200, based on the ARM920T it looks perfect for the job – now I just have to source it from an electronics company closeby and maybe find a development board for it, else I’ll have to figure out how to get it programmed manually. fun.


Back to reality, I’ve had to clean up my garage somewhat and I still need to order a semi-decent soldering iron and multimeter (probably from somewhere like rabtron.co.za), put up some shelving and source some organizers and bunches of components.

I’ve downloaded the datasheets and software (AVR Studio 4 and WinAVR) for my AVR STK500 and coded a little bit-runner.. the 8 LED’s simply flash down (in binary) to the number of the button you pressed.. quite simple, except for the weird peculiarity that the high 2-bits are on PORTC and the low 6-bits are on PORTB… I’m thinking there’s some kind of mis-wiring, if you ask me.. but anyhow, here’s my source code :

#include 

int main() {
	DDRB = 0xFF; // Output LED's
	DDRC = 0xFF; // Output LED's
	DDRD = 0x00; // Input buttons

	PORTB = 0xFF; // Set the default lights (0 = on and 1 = off)

	int pressed = 0xFF; // Set the default run value

	int gotoState = 0xFF;
	while (1) {
		// If a button is pressed, run to it
		if (PIND != 0xFF) {
			gotoState = PIND;
		}

		// While we're not at the last pressed button, run to it
		if (pressed  gotoState) {
			pressed --;
		}

		// Wait a little, else everything happens instantly!
		__builtin_avr_delay_cycles(500000);

		// Update the LED's according to the pressed value.
		PORTC = 0xFF & (pressed >> 6);
		PORTB = 0xFF & pressed;
	}
}

Going forward, I’m going to start sourcing all the components for my end-project, start doing a little more formal design-work (so that this becomes a lot less of a blur) and hunt for a victim lone calculator whose LED screen I can steal.. maybe I’ll get myself a full little LCD, but I’m not sure how much I can fit into 4k – i’ll have to figure out how to extend the execution size somewhat, i think, but lets see what pans out..

Till next time!

Phun Challenge: Factory

First off: whoever hasn’t got Phun – download it here! It’s a very cool 2D physics simulator.

I built an interesting challenge in Phun last night,  Download it here, and now I’m curious to see what solutions people could come up with to make this work, so I’ve decided to issue it as a competition with a small reward of R200 (approx $26) for the best solution.

Objective : Successfully and completely assemble as many of the 5 objects as illustrated in the orange zone example and place them in the demarcated delivery bays. This means you have to move the original pieces to the bays, they have to be attached together correctly and be placed right-side-up.

The simplified rules are :

– Don’t touch yellow area
– Of the existing items you only modify RED gear
– Don’t try looking for loopholes. 😛

The extended rules are :

– You may not work in, or modify anything in the demarcated yellow areas – Those areas have to be exactly identical in your solution as the original state.
– You may not move or modify any existing items/hinges except the RED gear.
– You can create whatever you wish and attach to the background or floor plane.
– At no point is your solution allowed to interfere with the orange (example) area.
– I will be the sole judge of the winner, so you’ll be disqualified if I consider something cheating 🙂
– If you submit multiple entries, I’ll only be testing the last one I receive.

Update: No closing date – if you’re the first to send me a solution, I’ll edit this post and you get the R200 (via pocit – www.pocit.co.za)

After the closing date, I will choose a winner on the best, valid submitted solution – A solution is valid if it gets at least 1 properly assembled object into a delivery bay at least once in the 10 times I run it.

I will be scoring each entry based on the amount of fully correct objects are assembled and placed correctly into the delivery bays over 10 tries (getting all 5 in all 10 runs = max score) – if there is more than 1 solution that can do that, I will tie-break it by the entry that is the most amazing and goes the extra mile – I favour faster, cleaner and more stable solutions.

Good luck and have fun!

%d bloggers like this: