In “Hierarchical Storage Management, from drive to cloud” I wrote about the idea of applying HSM by using a central storage or Cloud model. Here I discuss the next stage, using Peer-to-peer technology to create private clouds. The central remote cloud services are then useful as the expensive slow end point of data migration.
Hierarchical storage management (HSM) is a system of using layers or tiers of storage resources to migrate files according to various criteria. Data would autmatically be migrated from RAM to Solid-State Drives, to disks, and finally to Tapes. This is an old technology, around since the 1970s, and primarily used at enterprise business. However, various forms of HSM have been used in consumer products, services, or at the OS level. One example is a new service such as Bitcasa.
The use of P2P to synchronize storage to one’s own devices got a big boost with BitTorrent Sync (BTSync) created by BitTorrent. BTSync, though not a replacement for something like DropBox, offers an additional model to offset some limitations of cloud storage.
BTSync showed that Peer-to-peer (P2P) sync works and is useful. Could this approach be used as a base to provide not only synchronization but migration of files to different tiers? A peer-to-peer-hierarchical-storage-management (p2pHSM).
With this approach, the private storage hierarchy is a true cloud, a private network of storage resources. This private cloud could also be connected to traditional cloud storage vendors, like BitCasa or DropBox, creating a hierarchy of storage Clouds. A further optimization is for the private P2P HSM to arbitrate or bid with multiple external Cloud providers for best rates and other criteria.
Your extended family and real friends provides many computing devices to the private cloud: smartphones, NAS, USB drives, STBs, vehicles, laptops, PCs, tablets, and so forth. Each of these devices has limited storage capacities and bandwidth limits.
The storage resources on each device allocate a percentage of storage and bandwidth to the private cloud. The storage is secure and private, and its content available via access control permissions (ACL). For example, children cannot access their parents content.
- Your walking outside wearing your head mounted device, like Google Glass, and a Hummingbird flies into view, you turn on Record and create a video.
- The video gets transmitted to your mobile device.
- Your device notes that you are running out of space and a P2P storage request is made to the network of devices available.
- A high-end mobile phone is found that has extra space on a memory card.
- The original mobile device sends the video to that device.
- A week later, no one is viewing that video anymore so the system migrates that video to other storage.
- The next level is the laptop since it has plenty of room and is faster.
- A month later the video is again migrated to the family PC which has plenty of free space.
- Two months later it is again migrated to the family’s external cloud storage service if the monetary rules configured allow it.
At every file migration, the system seeks the optimization of price, performance, access, security, and capacity. Only when the private cloud is approaching certain limits or for backup purposes is the top, slowest, and most expensive tier, the conventional Cloud, used.
Of course, the history of file sharing systems is full of controversy. Some would say that a private P2P adds to the problems, notwithstanding that there are already plenty of public P2P networks and massive file sharing already occurring. Some issues are:
- Copyrights and fair use
- Is data accessed at the “Migration Level”, or migrated back?
- Private peer-to-peer
- Home-brew automatic tiered storage solutions with Linux? (Memory -> SSD -> HDD -> remote storage)
- Hierarchical Storage Management, from drive to cloud
- How to Make Your Hard Drive Infinite
- Hierarchical storage management
- Storage virtualization
- Data Tiering By Storage Location
- Wireless ad hoc network
- Using Git with BitTorrent Sync
- ZFS and the Hybrid Storage Concept