Private P2P HSM, a network for optimal virtual storage

In “Hierarchical Storage Management, from drive to cloud” I wrote about the idea of applying HSM by using a central storage or Cloud model. Here I discuss the next stage, using Peer-to-peer technology to create private clouds. The central remote cloud services are then useful as the expensive slow end point of data migration.

Hierarchical storage management (HSM) is a system of using layers or tiers of storage resources to migrate files according to various criteria. Data would autmatically be migrated from RAM to Solid-State Drives, to disks, and finally to Tapes. This is an old technology, around since the 1970s, and primarily used at enterprise business. However, various forms of HSM have been used in consumer products, services, or at the OS level. One example is a new service such as Bitcasa.

P2P Synchronization
The use of P2P to synchronize storage to one’s own devices got a big boost with BitTorrent Sync (BTSync) created by BitTorrent. BTSync, though not a replacement for something like DropBox, offers an additional model to offset some limitations of cloud storage.

BTSync showed that Peer-to-peer (P2P) sync works and is useful. Could this approach be used as a base to provide not only synchronization but migration of files to different tiers? A peer-to-peer-hierarchical-storage-management (p2pHSM).

With this approach, the private storage hierarchy is a true cloud, a private network of storage resources. This private cloud could also be connected to traditional cloud storage vendors, like BitCasa or DropBox, creating a hierarchy of storage Clouds. A further optimization is for the private P2P HSM to arbitrate or bid with multiple external Cloud providers for best rates and other criteria.

Scenario 1
Your extended family and real friends provides many computing devices to the private cloud: smartphones, NAS, USB drives, STBs, vehicles, laptops, PCs, tablets, and so forth. Each of these devices has limited storage capacities and bandwidth limits.

The storage resources on each device allocate a percentage of storage and bandwidth to the private cloud. The storage is secure and private, and its content available via access control permissions (ACL). For example, children cannot access their parents content.

  1. Your walking outside wearing your head mounted device, like Google Glass, and a Hummingbird flies into view, you turn on Record and create a video.
  2. The video gets transmitted to your mobile device.
  3. Your device notes that you are running out of space and a P2P storage request is made to the network of devices available.
  4. A high-end mobile phone is found that has extra space on a memory card.
  5. The original mobile device sends the video to that device.
  6. A week later, no one is viewing that video anymore so the system migrates that video to other storage.
  7. The next level is the laptop since it has plenty of room and is faster.
  8. A month later the video is again migrated to the family PC which has plenty of free space.
  9. Two months later it is again migrated to the family’s external cloud storage service if the monetary rules configured allow it.

At every file migration, the system seeks the optimization of price, performance, access, security, and capacity. Only when the private cloud is approaching certain limits or for backup purposes is the top, slowest, and most expensive tier, the conventional Cloud, used.

Of course, the history of file sharing systems is full of controversy. Some would say that a private P2P adds to the problems, notwithstanding that there are already plenty of public P2P networks and massive file sharing already occurring. Some issues are:

  • Security
  • Privacy
  • Copyrights and fair use
  • DRM
  • Is data accessed at the “Migration Level”, or migrated back?


What is Cloud Computing?

Someone from the non-technical business side asked me what is this Cloud computing all about. If you search for the term you find a bunch of definitions and descriptions.

Someone from the non-technical business side asked me what is this Cloud computing all about. If you search for the term you find a bunch of definitions and descriptions. For example, on Wikipedia:

“Cloud computing refers to the delivery of computing and storage capacity as a service to a heterogeneous community of end-recipients.”

That doesn’t sound like anything new. We had similar things even in the sixties. You have to read more of the above article to see the differences. But, to see what it really means a blog post, “Startups are Creating a New System of the World for IT“, puts it in context.