Private P2P HSM, a network for optimal virtual storage

In “Hierarchical Storage Management, from drive to cloud” I wrote about the idea of applying HSM by using a central storage or Cloud model. Here I discuss the next stage, using Peer-to-peer technology to create private clouds. The central remote cloud services are then useful as the expensive slow end point of data migration.

Hierarchical storage management (HSM) is a system of using layers or tiers of storage resources to migrate files according to various criteria. Data would autmatically be migrated from RAM to Solid-State Drives, to disks, and finally to Tapes. This is an old technology, around since the 1970s, and primarily used at enterprise business. However, various forms of HSM have been used in consumer products, services, or at the OS level. One example is a new service such as Bitcasa.

P2P Synchronization
The use of P2P to synchronize storage to one’s own devices got a big boost with BitTorrent Sync (BTSync) created by BitTorrent. BTSync, though not a replacement for something like DropBox, offers an additional model to offset some limitations of cloud storage.

BTSync showed that Peer-to-peer (P2P) sync works and is useful. Could this approach be used as a base to provide not only synchronization but migration of files to different tiers? A peer-to-peer-hierarchical-storage-management (p2pHSM).

With this approach, the private storage hierarchy is a true cloud, a private network of storage resources. This private cloud could also be connected to traditional cloud storage vendors, like BitCasa or DropBox, creating a hierarchy of storage Clouds. A further optimization is for the private P2P HSM to arbitrate or bid with multiple external Cloud providers for best rates and other criteria.

Scenario 1
Your extended family and real friends provides many computing devices to the private cloud: smartphones, NAS, USB drives, STBs, vehicles, laptops, PCs, tablets, and so forth. Each of these devices has limited storage capacities and bandwidth limits.

The storage resources on each device allocate a percentage of storage and bandwidth to the private cloud. The storage is secure and private, and its content available via access control permissions (ACL). For example, children cannot access their parents content.

  1. Your walking outside wearing your head mounted device, like Google Glass, and a Hummingbird flies into view, you turn on Record and create a video.
  2. The video gets transmitted to your mobile device.
  3. Your device notes that you are running out of space and a P2P storage request is made to the network of devices available.
  4. A high-end mobile phone is found that has extra space on a memory card.
  5. The original mobile device sends the video to that device.
  6. A week later, no one is viewing that video anymore so the system migrates that video to other storage.
  7. The next level is the laptop since it has plenty of room and is faster.
  8. A month later the video is again migrated to the family PC which has plenty of free space.
  9. Two months later it is again migrated to the family’s external cloud storage service if the monetary rules configured allow it.

At every file migration, the system seeks the optimization of price, performance, access, security, and capacity. Only when the private cloud is approaching certain limits or for backup purposes is the top, slowest, and most expensive tier, the conventional Cloud, used.

Of course, the history of file sharing systems is full of controversy. Some would say that a private P2P adds to the problems, notwithstanding that there are already plenty of public P2P networks and massive file sharing already occurring. Some issues are:

  • Security
  • Privacy
  • Copyrights and fair use
  • DRM
  • Is data accessed at the “Migration Level”, or migrated back?


Future of Google Glass: Continuum TV series

The TV series Continuum starring Rachel Nichols has some fancy high tech augmented reality from the year 2077. There are no nerd glasses, the corporate state actually does something to the eyes and other neural circuitry that is used in conjunction with a fancy super-computer skin tight (of course) costume. Pretty neat. The tech is not new, SciFi literature has been using these ideas for a very long time already.

To the naysayers of the Google Glass type of devices: If Science Fiction and other forms of imagination are any judge, we haven’t seen anything yet. The downside is that it seems privacy and security issues will not keep up with tech advances.

Alas, the TV series was cancelled (?). Yet, ‘Two and Half Creatures’ is still on. Go figure.