Using BitTorrent for large software deployments?

Oh well, its easy to come up with ideas, making them work is hard.  Even harder is making people get over the “but syndrome”.

I just (8/1/2010) saw this article on “Murder: Fast datacenter code deploys using BitTorrent”.

I know a lot of people always say I thought of this or that. Well, I thought of that. I brought it up when I was working at a start-up during the 2000 time frame.  I and a co-worker also thought that it would be useful in Enterprise Content Integration for actual data.

Just last year I looked into the use of Rsync and even a Distributed Version Control System like Mercurial or Git for a bandwidth deployment problem.  So many options.  Oh well, its easy to come up with ideas, making them work is hard.  Even harder is making people get over the “but syndrome”.

“New ideas pass through three periods: 1) It can’t be done. 2) It probably can be done, but it’s not worth doing. 3) I knew it was a good idea all along!”
— Arthur C. Clarke


  • Dec 21, 2013: Since the above has been written a lot of stuff has occurred in the IT industry, of course. Most relevant, BitTorrent came out with “BitTorrent Sync”. This product seems very applicable to software deployment. However, in an enterprise a more centralized and managed process would drive the process which could include the use of P2P syncing of resources.
  • March 9, 2014: Another great example: Sync Hacks: How Angie’s List Reduced Their Web Deployment Time to Seconds


Synergistic Social Agent Network Cloud

Multi-agent systems embedded in the social cloud offer more value then ‘Apps’


Multi-agent systems embedded in the social cloud offer more value then ‘Apps’.

Key words: agents, multi-agent, dynamic networks, social networks, Recommender system, cloud, cloud agent

SynergisticMobileAgentSocialNet MindMap
SynergisticMobileAgentSocialCloud Mind Map (click for larger view)


2014-10-28: I see Google is doing a new “Physical Web” effort”. Intro is here.


Mobile apps have not been very gratifying. Testing an app last year gave some clarity to what I felt to be a problem with the current App ecosystem. And, this is not just a mobile issue, but also for traditional computing platforms. I have been thinking of this subject for years. This is just, finally, a very simple and pragmatic example.


Last year I downloaded an app that locates the cheapest gas based on my current location. Whether cheap gas should be used in one’s car is not the point here. The app could have been one for finding the best licensed massage therapist or bookstore. The point, is this using mobile computing to its full potential?

What if the cheap gas station is located in an area where crime is very high? Should I risk a carjacking just to save 3 cents? What if I’m about to run out of gas now, is the cheapest gas too far away? We can get even more complicated of course. What if I have to be at an appointment, shouldn’t the cheap criteria be augmented with route info; the cheapest gas is the one easiest to get to on my way to or from my appointment.

In short, the current app is one-dimensional. Real life is multidimensional and the human brain easily makes decisions within this mostly analog fuzzy chaos. If an app cannot make decisions or recommendations in that same world, it collapses the dimensions, it is a dumbing down.


How can the app be made more dimensional? AGENTS. The app should really be an Agent that cooperates with other agents to fulfill a need, in this case finding cheaper gas. Thus, it should talk to other autonomous agents, such as:

  • calendar
  • law enforcement to grade destination
  • vehicle network for fuel requirements
  • traffic
  • mapper
  • GPS
  • weather
  • retail for
    • quality
    • complaints
    • hours of operation
    • costs
  • Social Net
  • Politics
  • Financial
  • map routing, and so forth.

It should also be informed by human agents in a trusted relationship with the user. What we then have is An Ad Hoc Dynamic Network of Social Agent Recommenders (AhDyNoSAR).

The Mind Map Diagram shown previously gives a contextual view of this idea.

Let’s look at another example. Someone is walking in neighborhood that has a few restaurants. The embedded Agent notes that the last time the person ate was a few hours ago (based on shopping venue, Calendar, etc.). The shop’s agents are contacted and a decision processing workspace is created. Is the person currently viable, do they have cash or credit available? Each store will check inventory and accounting ratios, does it need to offer a discount or promotion to this person? More agents mobilize to assert their criteria. What are the person’s tastes, dietary restrictions and allergies, past intake (who wants pizza twice in one day?), and other multidimensional agents in a problem space hierarchy are evoked.

After all agents complete their reckonings and the spontaneous net reaches a stable resonance, the person’s intimate personal soft computing agents make a decision. It turns out that the person is currently following their spiritual observance and is fasting today. This result is sent into the local agent milieu and starts a new search for resonance, so no food, how about some clothing or reading material? Again a new recommendation graph is created, religious and political leanings are queried, clothing and accessory rules are fired, ah, that is a very old turban, here are some suggestions.

Unfortunately, the person has now walked into a new map space, a neighborhood park. Now new agents awake: social engagement, entertainment, sexual, defensive.


It would be so gross if the information that this new cloud offers is shown as ads. A better approach is that this information space is entered as a virtual world, using technologies like that of Massively multiplayer online role-playing game (MMORPG). The consumer becomes an Avatar moving through Recommendation Space, a superimposed view on current locality based environments. Instead of or in addition to other consumers, the other characters are the various agents most visible recommendation goal.


Unlike Apps an Agent should always be considered adversarial. That is, even when an agent provides a benefit, it also can allow intentionally or via weaknesses a loss of security and privacy since it must negotiate information with other agents. Thus, though current or future standards may be used, they must be in virtual application spaces that use encrypted anonymous data. This will be just as virus and other malware, an ongoing battle.


It would not be optimal to require a download of an agent to each user’s location or device. Instead, agents will exist in the cloud as a multi-agent system. A user will have a private cloud virtual machine and address space for agent storage and recommendation space. To handle disconnected use, an agent will have a mobile agent shadow. It will provide simple assistance and will punt decisions and actions it cannot handle until connection to the cloud is established.


With Apps, the app provider may require purchase or try to enforce lock-in or an advertising monopoly. This can also be accomplished by centralizing the App marketplace. This may not work directly with Agents. Agents may not even provide an obvious visible function. For example, an agent may just contribute parking meter locations and status to other agents that use a map agent.

In the real world eventually someone has to pay the piper. So too will the development and use of agents must be rewarded. Some options are:

  • Advertising:
    An agent can contribute to an advertising stream that ultimately reaches the consumer facing user interface device.
  • Agent micropayments:
    Agents will negotiate among their collaborators to maintain a balance of payments, an agent of agents, and this payment is satisfied by the user or the user’s fee structure that the network provider maintains.
  • Purchase:
    The consumer will purchase agents. If the fidelity and number of agents is adequate the quality of service is greater.
  • Other.



Of course, the internet is currently wide open and thus this opens up predation to another level if Agent “sandboxes” are porous, if personal data is not secure.


The present cavalier attitudes regarding personal privacy exhibited by the large Internet service providers is a big warning sign that giving agents access to even more information would be just another data mining delicacy ripe for exploitation.


And now for an even more far out scenario. In a classic Science Fiction novel, before a character dies, a copy of their knowledge is captured. This intelligence is then available for implantation into someone as an “Aspect”, an agent that can add its unique expertise and judgment to the human host. That is a more radical direct means for accomplishing something that the social networking may evolve into, a means to collect knowledge and translate that into a ubiquitous intelligence.


Presented was a critique of conventional app centric mobile computing and a suggestion that Agent technology can provide a more realistic computing environment. The term Agent was not defined here. Perhaps the difference with an App is just intent or where the output is ingested. The experts are still debating Agent technology and its applications.



Further Reading

All rights reserved. No part of this document may be reproduced or transmitted in any form by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from Josef Betancourt.

This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License, See:

go Top

MercurialEclipse error 'dotencode' not supported!

on an Eclipse project I got this Team error: abort: requirement ‘dotencode’ not supported!

When using Team Mercurial to refresh an Eclipse project I got this error:   “abort: requirement ‘dotencode’ not supported!”

Strange I just updated the Intland Software’s MercurialEclipse feature and it’s running Mercurial 1.7.0.


Anyway, to fix I just went into the

plug-ins configuration: Windows -> Preferences -> Team -> Mercurial and unchecked the “Use default (built-in) Mercurial executable”.

I specified “C:Program Files (x86)TortoiseHghg.exe” or whatever is the location of the latest hg.

It worked. Hope it doesn’t break something else.


  • 2011MAR04: I updated to new Hg ver 1.8 and TortoiseHg 2.0. Had to make the change again in Eclipse. TortoiseHg 2.0, btw, looks beautiful. Nice work on the UI.

Further reading


Ralph Towner — Witchitaito

Cascading Configuration Pattern

Load configuration files in cascade to simplify and reduce excess data.


Many existing systems load configuration in cascade to reduce the use of duplicate properties and allow fine grained customization. This post expresses this common usage into a design pattern.

Keywords:   SCM, CM, Properties, Groovy, Config, CSS

Content:  Context, Forces, Solution, Consequences, Implementation, Code Example, Related Patterns, Related Intellectual Property, Further Reading


A business system uses property (configuration) files to configure particular environments or subsystems.   Many environments share the same properties and values, however, some are different and crucial.  To avoid missing any properties, all the properties are duplicated in each file, and required differences are changed appropriately. 

For example, if there are 100 properties required and there are 23 possible environments, that is 2300 lines of source to manage.  If there are any duplicated properties that do not vary between environments, then there is an opportunity to simplify the configuration system.  In this case, if we make only a “root” file have the full set of properties,  the total size is given by: 

T = L+npL; 
    T is total size, 
    n is number of files, 
    p is percent of each file that is overridden.
    L is number of properties  

Here the value is 560=100+23*.2*100). The reduction over the duplicated data is 75%.


There are many issues with having the same properties in multiple locations.  One obvious disadvantage, is adding a new property would require changing multiple files.  Also the files are larger and not really cohesive.   Tracking errors is also complicated, especially run-time errors due to configuration settings.  Any solution should not add its own complexity.  Thus, we should not require new types of configuration files and requirements.

Even when configuration could be isolated or systems modularized, there may be a need to have duplicated properties or reassignment to satisfy software development life cycle (SDLC).  A system will be different in dev, test, and production environments.


A hierarchy of property sources and a system that can load each source and override properties at a prior level will provide a partial solution.  This is the approach already taken in many systems.  For example, in software applications and servers, the final properties used are composed of those taken from various standard locations in a particular Operating System host.  In Windows for example, the Environment Variables are composed of those found in System and User name space.  In software tools, such as Mercurial DVCS, there is a command to show the results of the final configuration after all configuration sources are loaded: showconfig   show combined config settings from all hgrc files. In Git, one executes: git config -l.

Many build systems such as Ant and Maven, allow and expect the cascading of property files.  Note that in Ant, the inverse policy is used, the first property assignment wins.

And, of course, this cascading is seen in the Cascading Style Sheet (CSS) technology.


There is now the requirement that the cascade order is well known, non-variable, and robust.  Any intervening file must be loaded, else the system may destabilize.  Adding a new property or changing a property will require extra care since an environment or subsystem may be effected if the file contents are not properly designed beforehand.

To provide management and debugging support the implementation should provide traceability of configuration.  Thus, during use, the system must report (log) what was overridden, not changed, what is missing, etc.


Using this pattern is very easy.  One just has to determine how the system handles reassignment in the configuration loader subsystem.  If it allows resetting then the file hierarchy is from global to specific.

Where complexity may come is when one wants to satisfy more sophisticated requirements.  For example, a property may need to be appended.  In OS paths, for instance, paths are combined to create the final path.  Thus, the operations on properties could be a combination of:

  • create: initial assignment of a property
  • update:  a new value is assigned
  • delete:  the property is removed
  • append: append to existing value
  • merge:  use a pattern to merge a new value
  • locked: allow no change
  • fail on replace:  if a value already assigned throw an exception.

Code Example

In Java the java.util.Properties class will overwrite an existing property with the same key value.  Thus, to cascade properties, one just reuses the same instantiated Properties Object via the various load(…) methods.

Note that in some frameworks, like Ant, the reverse is true, properties do not get overwritten. So, the “root” properties must be loaded last in a cascade sequence.

Shown in listing one is a simple implementation written in Groovy, a dynamic language on the JVM.  The PropCascade class extends Properties and adds methods to load a list of sources.  The first source in the list is the root of the “cascade”.  In addition, a few methods explicitly specify the root source. From a traceability point of view, reusing java.util.Properties class may not be optimal. It gives no access to low level capture of the actual “put” action. Even with the use of Aspect Oriented Programming, with for example the AspectJ language, there is no joinpoint available to add it.

listing 1

 * File: CascadedProperties.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt

import java.util.List;
import java.util.Properties;

 * An extension of Properties that adds Convenience
 * methods to load lists of sources.
 * @author jbetancourt
class CascadedProperties extends Properties {
	//private Properties rootProperties = new Properties();
	//private boolean firstWins = true;
	//private boolean failOnDuplicate = false;
	//private boolean isTrace = false;

	 * Load a list of properties sources.
	 * @param list
	public void load(List list) throws IOException, IllegalArgumentException {
		list.each {

	 * Explicit file path is specified.
	 * @param path
	public void load(String path) throws IOException, IllegalArgumentException {
		load(new File(path).newInputStream());

	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 * @param inStream
	 * @param list
	public void load(InputStream inStream, List list) throws IOException, IllegalArgumentException {
		load inStream
		load list

	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 * @param reader
	 * @param list
	public void load(Reader reader, List list) throws IOException, IllegalArgumentException {
		load reader
		load list

	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 * @param path
	 * @param list
	public void load(String path, List list) throws IOException, IllegalArgumentException {
		load path
		load list

} // end of CascadedProperties

In listing two, the JUnit test class is shown.
listing 2

 * File: CascadedPropertiesTest.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt

import java.util.List;

import org.junit.Before;
import org.junit.Test;

import groovy.util.GroovyTestCase;

 * Test the {@link CascadedProperties} class.
class CascadedPropertiesTest extends GroovyTestCase{
	private CascadedProperties cp;

	/** excuted before each test method run */
	public void setUp() throws Exception {
		cp = new CascadedProperties();

	public void testloadListPaths() throws Exception {
		List list = new ArrayList();
		list.add path1
		list.add path2



	public void testloadListReaders() throws Exception {
		List list = new ArrayList();
		list.add reader1
		list.add reader2



	public void testloadListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream1
		list.add iStream2



	public void testloadStreamAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3



	public void testloadPathAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3



	public void testloadReaderAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add reader2
		list.add reader3



	public void testPutAgain() {
		cp.put("k1", "v1")
		cp.put("k1", "v2")
		assertEquals(cp.get("k1"), "v2");

	public void testLoadOneFilePath() throws Exception {

	public void testLoadTwoFiles() throws Exception {


	// class fields
	String path1 = ""
	String path2 = ""
	String path3 = ""
	File file1 = new File(path1)
	File file2 = new File(path2)
	File file3 = new File(path3)

	InputStream iStream1 = file1.newInputStream()
	InputStream iStream2 = file2.newInputStream()
	InputStream iStream3 = file3.newInputStream()
	Reader reader1 = file1.newReader()
	Reader reader2 = file2.newReader()
	Reader reader3 = file3.newReader()


Property Files Analysis

It is very difficult to eyeball a set of property files to see if there are many duplicates. I guess one could do some magic one line script that concatenates, sorts, prunes, etc. Here is an alternative using the Groovy language.
listing 3

 * Script: AnalyizePropertyFiles.groovy
 * Author: J. Betancourt
import java.util.Hashtable;
import java.util.Map.Entry;

 * @author jbetancourt
class AnalyizePropertyFiles {
	// these could have been created dynamically by reading the target folder.  But, that may pick up non-used files.
	static fileNames = []

	// where are the files?
	static basePath = ""

	// run analysis....
	static main(args) {
		def propList = [] // each property file will be load into a cell which contains a properties object.

		// put each file into properties object in propList.
		fileNames.each {
			def p = new Properties()
			p.load(new File(basePath + "" + it).newInputStream())

		// put all properties in one allProps object.
		Properties allProps = new Properties();
		propList.each {
		}// each propList
		def result = []
		int totalCount = 0
		// get how many times each property is used in all properties object
		allProps.each { prop ->
			String key = prop.key
			String value = prop.value
			def values = []
			int count = 0
			propList.each { p ->
					def curValue = p.get(key)
			} // each property file
			StringBuilder vb = new StringBuilder()
			values.each{ s ->
				vb.append("[" + s + "], ")
			int numberValues = values.size()

			StringBuilder sb = new StringBuilder()			
			sb.append(count).append(",").append(values.size).append(", ").append(key).append(", ").append(vb.toString())
			totalCount += count				
		} // each allProps property
		println("Total number of times properties are repeated: " + totalCount)
	} // end main

} // end class

Related Patterns

Related Intellectual Property

Cascading configuration using one or more configuration trees“, U.S. Patent number 7760746, 30Nov2004,

Further Reading

What is Eclipse's Current Workspace?

Unless you start Eclipse with the “-showLocation” option there is no way to know what the current workspace location is?

Maybe someone knows the answer. Unless you start Eclipse with the “-showLocation” option there is no way to know what the current workspace location is. Another method is to use the Eclipse menu system: File -> Switch workspace -> Other. The workspace will be in the resulting dialog box.

There are also a few configuration settings available in the Eclipse -> About -> Show configuration:

One reason to know is that when I go to create a new project, do I specify the workspace or is the current workspace the one I want? I don’t remember if the workspace is shown or specified on project creation, will have to check.

Maybe I’m missing some internal Eclipse reason for this (similar to that crazy bug when the copyright name was changed inside Java) and there is no way for Eclipse to know this. Still, seems lame.

10 JUL 2012: Eclipse Juno: -showLocation doesn’t seem to be working, doesn’t show location in title bar of Eclipse.
17 JAN 2011: Found the answer, see below. I was on the right track. If I read the FAQ entry correctly, the current project is accessible (even though there can be many active projects), it is just not exposed in the Eclipse UI? But, that doesn’t really show the workspace, of course.

Further reading

Maria Pia de Vito with John Taylor & Ralph Towner – I Knew It was you

Setup SSL Issues On Tomcat

Get errors running SSL on Tomcat. Found one solution.

I was trying to setup an external HTTP server so that I can test SSL connections from my Java app. I’ve done this with Tomcat server before. It was easy, just follow the instructions on Tomcat’s document site. All done.


  • Windows XP Professional
  • Tomcat 6.0.29
  • Java 1.6

Not this time. A host of problems:

1. PKIX path building failed: unable to find valid certification path to requested target

2.  SEVERE: Error initializing endpoint
java.lang.Exception: No Certificate file specified or invalid file format
	at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method)

3.  SEVERE: Error starting endpoint
java.lang.Exception: Socket bind failed: [730048] Only one usage of each socket address (protocol/network address/port) is normally permitted.  
    at org.apache.coyote.http11.Http11AprProtocol.start(
    at org.apache.catalina.connector.Connector.start(

The last issue above is very weird. I ran netstat and no other apps are using the ports that Tomcat is trying to use.

I won’t bore you with the details, suffice to say, I just removed the native library, tcnative-1.dll, from the Tomcat folder and everything works.

Well, not everything, I’m still getting, PKIX path building failed: unable to find valid certification path to requested target

But, that is more tractable.

Modular Continuously Shippable Applications

This was not just a test-driven development (TDD) approach, I said. For example, we could use a Command Line Interface until we have the Rich Internet Application framework ready, we could use flat-files until we knew what the data model should be, etc. The “product” was in flux based on potential customer feedback. So, it was really to do the simplest thing that will get the application doing real work as soon as possible.

Long ago in 2000 at a job I suggested to the development teams that an approach we could use is to start on day one with a shippable product.  That is, using Stubs, Mocks, and fake data, we could create the complete app that is testable and shippable from day one.  Since the first customers are the investors and potential partners, they would get to see what we really intended to create up front, in working code, not fancy PowerPoint presentations.  The working code would have served as the presentation and the base on which to build the eventual product to ship to external customers.

This was not just a test-driven development (TDD) approach, I said. For example, we could use a Command Line Interface until we have the Rich Internet Application framework ready, we could use flat-files until we knew what the data model should be, etc.  The “product” was in flux based on potential customer feedback.  So, it was really to do the simplest thing that will get the application doing real work as soon as possible.

The advantages I thought could have been:

  • Faster to market
  • Better API (no fluff)
  • System always testable
  • Infrastructure for testing and QA
  • Usable for presentation
  • Modular

What was the response?  Nothing.  Oh well.  It would have been great to see what could have been accomplished.

I was reminded of the above as I stumbled upon “Architecture Review: Mock-Driven Three-Layer Architecture“ on James Shore’s The Art of Agile blog.

Now there has been much more creative work and new agile languages that would have made my idea more doable.  In fact, many of today’s successful frameworks are responses to the excess complexity that I had to deal with then.



“Potentially Shippable product – A Myth or Reality?”:

Continuous Integration:

“Learn how to Pivot — or you’re toast”:

Mock Object:

Mock-Driven Three-Layer Architecture”,

“Mocks Aren’t Stubs”,

“Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))”,


Why use an AOP language when other languages can now do Aspects?

Interesting research, JOT: Journal of Object Technology – How AspectJ is Used: An Analysis of Eleven AspectJ Programs: Sven Apel, on how an AOP language, specifically AspectJ, is used in a few sample projects. Was it useful, required? The jury is out.

Interesting research,  JOT: Journal of Object Technology – How AspectJ is Used: An Analysis of Eleven AspectJ Programs: Sven Apel, on how an AOP language, specifically AspectJ, is used in a few sample projects.  Was it useful, required?  The jury is out.

Interestingly the author correctly mentions that some languages now have rudimentary support for handling of basic crosscutting concerns.   And, since the analyzed projects did not really need advanced AOP techniques,

“numbers indicate that languages that provide only basic crosscutting mechanisms are appropriate to implement a large extent of the analyzed programs.”  

Note that the study does not apply to container based AOP as used in frameworks such as JBossAOP and SpringAOP.  Several Dynamic Languages use a Metaobject Protocol.

I used AspectJ for some utility and diagnostic help.  The language is fine and the concepts are approachable.  It’s the configuration, tools, and environment hassles that put a damper on things.   That’s why frameworks and tool support like that found in Eclipse are critical.

Adding Logging around all of the Methods of a Class with Groovy“, Ted Naleid.

Further Reading

JOT: Journal of Object Technology – How AspectJ is Used: An Analysis of Eleven AspectJ Programs, Sven Apel

McClean, John, “Painless AOP with Groovy“,

Meta-object Protocol,

Dynamic Programming Language,