SuperComputing Futures: the Next Sharing Paradigm for HPC Resources

The past few days, I have been reading Nicolas Dube’s PhD thesis [1] that proposes and discusses an economic model for HPC Resource trading. I found this thesis very interesting to read, and believe that I will be able to use at least some (if not many) of the ideas that are presented.

Small summary of the thesis
The first chapter of the thesis gives a general introduction into High Performance and Grid Computing. It also explains the fundamental ideas of the grid as a utility (utility computing), and how theories and ideas from economics can apply to this new way of viewing computer resources (HPC in particular). The chapter ends with the problem statement and further thesis outline.
The second chapter gives an introduction to finance and market economics; specifically the topics of commodity trading (using auctions) and its analysis. At the end of the chapter, the author links these concepts to grid resources and explains how by viewing these resources as commodities, the same economic market principles as used in other commodity markets can be applied to those grid resources.
In the third chapter, Dube proposes his actual market model (or grid exchange). Some of the key issues addressed in this chapter are how to specify resources in bids and offers when participating in a auction, the notion of Grid Credits and a price discovery mechanism for providers and clients.
In the fourth chapter, a lot of simulations are done to verify the model proposed in the third chapter. Some interesting results concerning market equilibrium and price stabilization are shown. Special interest is given to the bootstrapping of the proposed market model.
In the fifth chapter, some possible model optimizations are discussed, together with some other issues that should be considered when actually installing a grid exchange.
Finally, the sixth chapter contains an overview of the discussed material and achieved results.

Personal Ideas
Dube takes a theoretical approach to designing a grid exchange, using proven concepts from economics and finance. I believe that I can benefit from the results of Dube’s thesis by incorporating some of his ideas in a more real-world implementation.
For example, the ideas of Resources Sets to specify the needed resources of clients and resources offered by providers can easily also be used in a cloud setting.
Also, a price discovery mechanism is a necessary tool for providers and clients to have an idea of how to make intelligent offers and bids. Dube’s proposed mechanism is at least a good attempt to fill-in the theoretical logic used by such a tool.
Furthermore, I can surely benefit a lot the various other remarks and ideas spread throughout the thesis.

However, a ifference between Dube’s grid and the grid that we are considering is the fact that we are working with cloud resources that typically use virtualization in order to optimize resource usage. This primarily affects the providers as in the cloud setting they will typically offer different types of instances instead of barebone hardware (under utilization of a machine is less of an issue, since the left-over capacity of a server is used to schedule other VMs).
Note that it can be interesting to create a market that allows both types of resources to be offered (barebone hardware vs VMs), since in some HPC applications, the need for specific hardware platforms will always remain.

I have not given further thought to whether VMs can easily be integrated into Dube’s model, this might be the topic of an other blog post.

However, in general I think that it would be a good idea to combine some of the more theoretical ideas from Dube with some more practical cloud oriented ideas and algorithms from the GridEcon Project (together with some of my own ideas ofcourse ;-) )

I will post some more ideas about Dube’s Resource Sets within the next couple of days.

References
[1] SuperComputing Futures: the Next Sharing Paradigm for HPC Resources, Nicolas Dube, http://www.nic.qc.ca/Site/PhD_Thesis.html