I did give the interface some thought and decided to try and implement a kinetic scroller.
It's pretty difficult to find a definitive explanation of Kinetic Scrolling or Flick Scrolling, but I think this pretty much sums it up:
"Kinetic scrolling is the popular term to denote the scrolling of a long list with a bit of physics so that user feels like moving a wheel. Such a list view is then often referred as a flick list, caused the scrolling involves some sort of flicking gestures. " 
Apple have a tutorial showing how to use their implementation here 
The physics are an illusion, and the approach I used is simple.
- Click and hold the background to drag.
- When drag is released, note how far it moved since the last draw.
- Use that distance as the speed (or energy) of the scroll.
- Continue moving in the same direction, but start reducing the speed (decay).
- When the speed slow below a threshold, stop the scrolling.
The coloured bars are just for showing the movement clearly.