Python, Amazon, graphs, oh my!

I'm a regular reader of William J Turkel's blog, Digital History Hacks.
His recent posts (here and here) about analysing his course's reading list inspired some tinkering of my own.

In William's first post on the subject, he uses the ASIN (Amazon Standard Identification Number) from each of the books in his reading list and Amazon's API to request a list of similar books.
This generates a list of paired ASIN, up to 10 pairs per original title.

Original Title ASIN 1, Similar Title ASIN 1
Original Title ASIN 1, Similar Title ASIN 2
Original Title ASIN 1, Similar Title ASIN 3
Original Title ASIN 2, Similar Title ASIN 1
Original Title ASIN 2, Similar Title ASIN 2
Original Title ASIN 2, Similar Title ASIN 3

And as William shows these pairs can be used to construct a graph, which can be used to visualise the degree of connectivity of the reading list.

My code uses parts of Williams's so if it looks familiar, that's why.

First scrape the ASIN from the webpage, my version of this script is here scrape1.py.

So we have our list of paired ASIN, so let's query amazon and get some similar titles.
You'll need to get your own Amazon Web Service ID which is a trivial enough process.

I have a couple of simple calls to help extract the data we want, in a module called amazon.py.
The process is simple enough.

1. Load the pickled ASINs
2. Query Amazon
3. Compile a list of ASIN pairs
4. Pickle and save the list of pairs.
   

The code for that is here, get_similar1.py

I found this article, Python MiniDom, useful when playing with the XML data returned by Amazon, and theres a more comprehensive Amazon wrapper project for python, pyAWS.

OK, so now we have a list of pairs, we can start to play with Graphviz.
There's a lovely Python interface for Graphviz, pydot which I think is great.
Basically using pydot, graphs, nodes and edges become objects and this make it very easy to use.
Let's start by graphing all the original ASINs, and show links where they are similar to each other.

1. Load the pickled ASINs
2. Load the pickled ASIN pairs
3. Create a graph object
   
4. Create a node object for ASIN and add to the graph
5. Create an edge object for each linked original ASIN
6. Save the graph

Here's the code, graph1.py , and here the resulting graph.



It's quite clear that many of these books are considered similar by amazon, and some stand out much more than others as hubs in a network.

We can quantify that connectedness, and use the data to alter the graphs appearance.

If we count he number of nodes which connect to or from other nodes, we can apply that to the font size, which is often done in tag clouds.

To do this is quite simple, we'll use a dictionary object to assign a value to each ASIN, and as we cycle through the pairs we'll increment the value.

Then, when we create the node, we'll set the font size.

here's the code to calculate the weights:

# for each ASIN associate a value
weight={} for asin in asins: weight[asin]=0

# for each pair
for pair in pairs:
# only if they are one of the originals
if pair[1] in asins: # increment the weight
weight[pair[1]]+=1
weight[pair[0]]+=1

and here is where that value is used, and while we're at it let's make the nodes circles.

# add a node for each original title for asin in asins:
node=pydot.Node(asin, shape='circle', fontsize=8+weight[asin])
g.add_node(node)

the new code, graph2.py and the results...



The larger circles make quite a difference, but for some reason my system does have the right fonts. I think it still illustrates how by calculating and using the weight data we can make the over all picture clearer.
At this point the jpg files are getting large, and the quality is not that great. An alternative graphics format is Scalable Vector Graphics (SVG). Read about it at W3C and at Wikipedia. Firefox has native support for SVG and Adobe provide a viewer.

This format will provide nice fonts and smooth lines and curves, and some other useful features, which we'll get to later.

OK, let's use that weight value again, this time to add some color and view as an SVG.
I have a helper module called gradi.py, which has some functions which generate colour gradients, we can use this to calculate colours for our nodes.
Starting with yellow for node with no connections, and red for those with the most, everything between we'll make orangish.

First we define our colour range and gradient.

loColor=gradi.HTMLColorToRGB('FFCC00')
hiColor=gradi.HTMLColorToRGB('FF0000')
colourgradient=1.0/max(weight.values())

Now when we define our node, we'll calculate the colour and fill the circle.

color=gradi.RGBToHTMLColor(gradi.RGBinterpolate(loColor,hiColor,colorgradient*weight[asin])) node=pydot.Node(asin, shape='circle',style='filled', fillcolor=color, _ fontsize=8+weight[asin])

And then save as SVG

g.write(output_filename+'.svg',format='svg')

Here the modified script, graph3.py and the results, click the image or here for the SVG.
I recommend this Firefox Zoom and Pan extension for SVG files.



I prefer the look of the SVG files and the format provides some useful features that bitmaps can't, for instance we can have Graphviz add http links from the nodes and edges, and include tooltips too. Hovering over nodes will show the title, and clicking will open the Amazon product page.

To add this functionality we just query Amazon for the title and include that data as a node attribute.

node.set_URL('http://www.amazon.com/gp/product/'+asin)
node.set_tooltip(amazon.getelement(amazon.AmazonAPI(asin),'Title'))

Here the adjusted script, graph4.py and a link the new interactive SVG.

I think this is a great improvement over a bitmap of product codes.
In part two we'll add the titles which were suggested by Amazon.