Fortunately, on the RackSpace Cloud, the solution to this problem is simple – simply increase the maximum memory allocated to the site so that there is sufficient available during the auto-upgrade process.

Is all you need to do is add the following line to your .htaccess file in the root of your web site and the auto-upgrade will then work.

php_value memory_limit 64M

You can change the 64M value – theoretically 32M should work, but sometimes doesn’t. 64M worked for us, or you can increase in 16M chunks to find a value that works for you. Try to be sensitive to the operation of the Cloud and not hog lots of memory though!

Here’s what I have experienced and what I am doing to mitigate things. It would be interesting to hear your experiences (using any similar products) so feel free to post below.

For the past couple of months, there has been a nagging doubt about Live Mesh. Occasionally, a few files were continually flagged as having a synchronisation conflicts – meaning Live Mesh thought that the file had been updated on two separate devices (computers) in between separate synchronisations. This meant that Live Mesh kept multiple copies of the files so that I could manually merge the changes and ensure I had the latest version. However, despite doing a manual merge, Live Mesh kept insisting that there was still a conflict even though I knew for definite that there was not.

This problem sewed the seeds of doubt about the thousands of other files I had under the watchful eye of the Mesh.

I also discovered that some files – particularly my website source code – was not always being synchronised and some websites were missing from different devices in my Mesh. Sadly, this meant that I could not depend on Live Mesh for day-to-day work. I understand that Live Mesh is still beta, so there will be teething problems, but Microsoft’s recent suggestion that the Live Mesh system is going to be more about “plumbing” rather than direct customer facing tools suggests that there may be a move away from what we currently experience in the beta.

So, what to do? Back to the beginning …

My ultimate goal is to be able to access my files wherever I am, and ensure I am working on the latest copy of the file. I have two principle types of files I work on – documents and websites – and while the basic need is the same, the way they are managed is better separated into version control for the websites, and simple access for documents. This means that I need a virtual disk in the Cloud for most of the documents – which I already had with Live Mesh, but was instead relying on the flawed synchronisation functions – and a hosted version control system.

Since we work with Rackspace at Emissary and Rackspace have a range of Cloud offerings, we asked if they had anything that could integrate desktop access directly with cloud storage (there was nothing that enabled this with Live Mesh). The original plan was to link my desktop with the existing cloud storage available to Emissary through our existing contracts with Rackspace, but they mentioned Jungledisk (which, incidentally, Rackspace purchased earlier in the year) and a couple of other systems, and so I did some further research.

Jungledisk hosts the files in the Rackspace Cloud, and while it is currently US-based there are plans to roll out global data centres over time (which will mean quicker access from anywhere on the planet). Our experience as a company with Rackspace has been exceptional and the pricing model for Jungledisk was good – several Gb of files meant about USD3 per month and backup was provided. I trust Rackspace more than I do Live Mesh and decided to move my files from one Cloud to another. It took a while copying everything across (and resolving the conflict and missing file issues that Live Mesh had introduced), but I now have a complete working library in the Cloud which I can access directly from any computer with Jungledisk installed, or via a web interface.

The downside is that even with high-speed Internet, some large files can be slow to access at first, but the built-in local caching of Jungledisk takes care of this and file access is the same as if the file were local once you access it a couple of times.

The next challenge was to find hosted version control for the website code. For this I looked at hosted subversion (or SVN). There are a number of hosted SVN providers, and there are some free tools (e.g. TortoiseSVN) which provide integration between local PC-based code libraries and Cloud-based SVN. At present, I haven’t settled on one complete solution, but am currently leaning towards migrating to the Zend Studio IDE (away from NuSphere phpED) as it has integrated SVN management and works on a Mac, plus Beanstalk as a hosted SVN (because it sits on top of the Rackspace Cloud). Hosted SVN also has the benefit of allowing easier management and sharing of code between a virtual team of developers by creating a central company repository without needing to physically port/merge files or set up VPNs.

(If you have any experiences or recommendations for hosted SVN and associated PHP IDE tools, please comment below. Thanks.)

So what of Live Mesh?

I am still using it to sync my music library peer-to-peer. The music library as a whole does not change often, so I feel it is relatively safe to commit to Live Mesh. It is also easy to check if albums are complete on each device!

The end result is a hybrid of different solutions to manage different types of files as there is not one tool to rule them all (at the moment).

However, the post just told me there was a lot of debate about the issue and that a task force was working towards a definitive definition.

Not much use, so I thought I would add to the corpus of articles and definitions by publishing my (Emissary’s) definition of Cloud Computing.

Cloud Computing is the technology used for the provision of web-based resources (i.e. storage, compute cycles and bandwidth) for web-based activities (e.g. content delivery) to a third-party without the third-party needing to understand how the resources are provided or needing to worry that there are enough. To use an analogy, the provision of the service of Cloud Computing is like the provision of electricity – it is a utility in the true sense – because the user of the electricity does not worry how it is generated and is able to consume as much or as little as they need for their purposes. Cloud Computing, like electricity, is then billed in accordance with how much is used.

Because this is a relatively new term, it is often applied to everything and anything so that companies can sound like they are at the cutting edge of technology. The most commonly associated utility with Cloud Computing is Software as a Service (SaaS). In general, this is provided on a licensed basis which is usually calculated on a per-user, or per-client basis. While it is a utility, unless it is delivered from a fully-scalable, black-box server infrastructure that meets the definition of Cloud Computing as set out above then it is simply SaaS. Another service commonly associated with the Cloud is virtualisation technology because it provides a black-box service to the end-user. However, unless it scales and provides the flexibility that a true Cloud should, it is simply a means to abstract technology knowledge from the end-user.

The reason that it is easy to blur the line between technology such as SaaS or virtualisation and Cloud Computing is that they originate in the same place – on a server somewhere out there on the Internet. By the same token that a kettle or toaster are not simply called “electricity” we should be clear that a program (e.g. website or virtualisation tool) running on a server infrastructure is not Cloud Computing, but may use it.

For further reading, please read my previous article Cloud Computing vs Dedicated Servers.

Related articles by Zemanta

• Big Pharma Exploring and Moving Data to the Cloud – IT Execs Talk about How It is Being Done (ducknetweb.blogspot.com)
• Amazon offers more cloud control (news.cnet.com)
• Akamai Edges Into the Cloud, Surveys State of the Internet (xconomy.com)
• White House Leading Cloud Computing Charge (elasticvapor.com)
• Sun’s new mantra: Call us the ‘cloud company’ (news.cnet.com)

I spent some time scanning various forums to try to understand this problem better and a common thread emerged:

Uninstall Skype.

So, I did this (I had Skype 2.5) and Office installed without any errors. I then re-installed a later version of Skype and all was well.

Not sure why they interfere but it appears to have something to do with ports. Hope this solves your problem – add a comment to say thanks if it does

I was quite impressed with the simplicity of the interface – it was very informative about what was being uploaded to the cloud, what was being downloaded and what it was going to do next. It was also very easy adding existing folders to the sync set (much more so than Live Mesh: see Live Mesh Syncing Folders).

Everything was going well until Syncplicity told me that I had exceeded my storage quota and would need to pay to upgrade my account in order to continue to sync the files. This was strange because I only had 1.6Gb of files to sync, and allowing for possible differences in quota allocated to the files in the cloud versus the Windows PC, this should not have been an issue.

I investigated further and found that it was due to Syncplicity’s conflict resolution management system.

When Syncplicity finds a conflict it keeps the original copy of the file from that machine, plus creates additional copies of the file – 1 from each machine in the sync group (if there were 3 or more computers there would be 3 or more extra files) – but doesn’t tell you it has done this. This is a sure way to eat up the available storage. So, for example, I get:

• foo.php
• foo (edited by Edward Terry on Machine A @ 1-12-08 17:56).php
• foo (edited by Edward Terry on Machine B @ 1-12-08 12:36).php

Syncplicity then proceeds to synchronise these additional files across the connected computers and chews up the remaining space leading to the “you have exceeded your quota” message. Unfortunately, Syncplicity (at least as far as I could see from the user interface) offers no way of finding or managing these conflicts and you have to track them down in their respective directories manually.

The trouble with this is that my 1.6Gb is made up of thousands of small files in hundreds of directories, and so the task of tracking down and resolving these conflicts is impractical. In my case I had over 2000 such conflicts which resulted in over 4000 “edited by” files scattered around my directories and resorted to the DOS command “delete /s/f/q ‘*edited by Edward*’” which removed them all in one go. Not very elegant.

Syncplicity is still a beta at this stage, so – hopefully – they will sort out the synchronisation management issue. But for now I have stopped using it because of this issue as it creates more of a problem than it solves by using it (sorry Peter).

Back to Live Mesh.

Live Mesh doesn’t appear to keep shadow copies of files in the file system, and so doesn’t appear to use additional storage when conflicts occur. Instead it informs you there are conflicts in the news window alongside the synchronised folder when you open it. Since I have a handful of root folders which I synchronise it is quick and easy to check all conflicts across the 3Gb I currently sync with Live Mesh.

The down side with Live Mesh is that conflict management is tedious because you have to confirm what you want to do with each and every file conflict one by one, and Mesh keeps you waiting while it processes your decision before you can address the next conflict. It also only lets you process batches of 50 conflicts, so you can be there for a long time. OK, Mesh is still beta also, and the conflict management is more helpful than Syncplicity but it’s still tedious. (I current have over 11,000 files in my Holding Area directory of Live Mesh which suggests a very large problem).

According to the Mesh blog, Microsoft have a tool at their data centre which can be executed to help you clean up large-scale conflicts, especially those caused by tombstones. However, it’s not clear if this is an actual service you can request, or if it was a one-off when they did a specific upgrade to the Mesh software.

Ideally, what I would like to see out of a conflict management tool is something like this:

System Tray Notification:
There are 1863 file conflicts that need your attention. Click the bubble to view them.

When you click the notification a dialogue box pops up with a table of all conflicts. You then scroll through the list and select what you want to do with each conflict and then hit the “resolve” button and let the application process the results in the background.

The list should be a simple, interactive interface.

At the end of the background task, you would get another system tray notification telling you the task was complete and offering a report. This would show any failures or unresolvable conflicts for you to follow-up manually, but should result in the bulk (if not all) of the conflicts being resolved and allowing the user to get on with working and not lose hours to resolving conflicts one by one (as I have so far and will continue to do until I clear the backlog).

The difference engine in the synchronisation tool should be intelligent enough to check file contents and determine if there are any byte differences or if the files are exact copies. Conflicts that are only a difference in the date stamp on the file could be auto-selected to “keep latest” in the initial table of conflicts, while others that have differences would be left to the user to decide.

The synchronisation tool could be configurable to allow the user to select their own visual difference tool – I use the excellent open source WinMerge – so that they could view differences between files to make a decision and manage how to proceed. The sync tool could also have its own in-built basic visual difference tool for users who don’t use their own. This is OK for ASCII files, but more complex for binary files (although not insurmountable).

I still favour Live Mesh – despite the conflict issue – if only because it is more transparent than Syncplicity, and also has not yet deleted live files (as far as I can see so far) as SugarSync once did, and allows me to use my existing directories which DropBox does not.

I would like to see a simple and effective conflict management system in any and all of the cloud storage synchronisation systems because, as we commit more data to the cloud we will need effective ways in managing the little hiccups that occur along the way. Who will be first? (Or are any of the main players already claiming they can offer this?)

However, there is one feature I have struggled with as it’s not entirely intuitive – and that is synchronising folders between two (or more) computers and being able to retain the same directory path to the files on all machines.

This is something that SugarSync did well – it assumed the same path on connected machines. DropBox worked beneath its own equivalent of “My Documents” and achieved the same.

However, Live Mesh requires more user intervention to achieve the same results which is a tad frustrating.

In order for Live Mesh to correctly synchronise the same folder path on multiple machines you have to follow these steps:

• On Machine A, add the folder to Live Mesh BUT do not synchronise it with other machines in your Mesh (change the synchronisation option to “never for this device” in the dialogue box that opens when you add a folder to your Mesh).
• On Machine B, go to your “Live Mesh Folders” container that has been created. In there you will find shortcuts to both fully synchronised folders, AND greyed out (lighter blue) shortcuts to folders that have not been fully synchronised (ie the one you just set up on Machine A).
• Right-click (for WIndows users) on the greyed out folder and click “change sync settings”.
• (This is the only time that the current beta of Live Mesh will allow you to change the directory path for synchronisation, so you have to do it this way).
• Browse to the directory path you want this folder to be synchronised with and select/create the folder.
• Make sure you change when to synchronise this folder setting to include Machine B.

This will then replicate your original folder from Machine A to Machine B – or synchronise the data between the two machines if the folder already exists on Machine B also.

Remember that you can bypass the Live Desktop (the web mirror) at this stage by selecting “never for this device” against the Live Desktop. This will enable peer-to-peer synchronisation between computers and not store anything in the Cloud (useful if you have more than 5Gb of data to synchronise).

If you don’t follow the steps above you will find the folder from Machine A appear on the desktop of Machine B. There is no way to move it once this happens – the best way to resolve it is to log in to your Live Desktop online and turn off synchronisation with Machine B. This will convert the Mesh folder (blue) into a regular folder (yellow) and you can then safely delete or merge it. Do not delete it while it is still connected to your Mesh or the original files on Machine A will also be deleted!

It’s anoying that you have to go through this process and can’t simply move folders on Machine B, or set an option to tell the Mesh to replicate the folder structure, or change the mapping on-the-fly from the “Live Mesh Folders”.

I hope this feature becomes available soon as it will make replication much easier and intuitive, but for now we’ll have to resort to the manual process outlined above.

Back in 2001 when I set up my first web company, it was always my ideal that the company should be truly virtual and not be tied specifically to any physical office space. While it is important for people to come together to build ideas, this can be done in any space that provides the correct tools – which for us means whiteboards and notebooks.

I still value physical interaction with people for certain meetings as you get a visual feedback which you cannot get on the Web. You can see all of the participants’ body language, and it is much easier to be completely inclusive in such arrangements. Virtual meetings – using Skype and whiteboarding technologies – are OK in some situations, but not all.

However, these meetings do not make up the bulk of the day to day work – this is spent with the computer either writing, desigining or programming. And, as long as the computer is present with the right tools, this can be done anywhere. As long as the files are there also.

This is where the Cloud comes in (see also previous post: Cloud Computing vs Dedicated Servers).

My original intention was to embrace Cloud Computing and adopt an application to manage the replication and synchronisation of my business files across multiple computers – namely the office PC and the laptop – without actually needing to physically connect the two.

Initial reviews of available systems revealed two main contenders – SugarSync and DropBox. The former appeared a more mature product and with a wider functional base, so I decided this would be my synchronisation tool of choice – subject to testing it out of course. Ever the skeptic!

SugarSync

So, about a moth ago I installed SugarSync to provide a cloud-based store for all my work files. The set-up is simple – create an account online and then install the SugarSync client on all computers that need to be kept synchronised. SugarSync then ensures that all selected folders are replicated across all computers linked to that account so all your files are everywhere, as well as stored online in the Cloud. You can also configure SugarSync as a straight backup tool so that copies of the files are stored only in the Cloud but not replicated to each computer. Personally I opted for full synchronisation to make sure there is always a backup and I don’t need to be online when I need to access my files.

So how was this to change my life?

The main benefit would be that all my files would be accessible to me at any time – even from a mobile phone or any Internet connected computer. I no longer would have to depend on USB memory sticks to carry around a sub-set of my files and find myself caught out when somebody calls and I need something I don’t have.

This would make working much easier – I just open my computer and get on with work. A few minutes after I have saved the file (usually), SugarSync would back the file up in the Cloud and is then ready to send it to all the other computers I have connected to the account when they are switched on and the SugarSync client is running.

Was it easy making the transition? Essentially yes.

SugarSync set-up is easy – just create an online account and install the software.

I started with a sub-set of my files – about 5Gb of data – large enough to be a good test, but not committing everything. This took a few days to push up into the Cloud because upstream data rates are quite slow in the UK. Once the files were there, I installed SugarSync on the second computer (laptop) and started the download. This was much quicker and only took a few hours. I tidied up a few duplicates and manually merged in the contents of my memory sticks and let SugarSync sort out the synchronisation which it appeared to do without a hitch.

However …

There were some teething problems with the install on the Vista machine which I did not discover until a few days later. This required a re-install and created a swathe of files in limbo. SugarSync support is very slow and generally takes a few days to provide responses to support tickets. Also, there is no user manual explaining the basics – such as the “reclaim storage” option – and what to do in these situations. Nor could I find anything online. There is also no community support or forums. Consequently, in the intervening days there was a lot of faith held in the accuracy of the data held in the Cloud and what had actually been synchronised on my computers.

After three days I received a response which “explained” what the “reclaim storage” option was about. Unfortunately, the response didn’t really clarify the situation and after a couple of weeks trying to resolve it I resorted to gambling with SugarSync and trying out something that could have been easily addressed.

Here is a summary for anybody experiencing a similar dilema with reclaim storage:

SugarSync explanation:

The files shown under “Reclaim Storage” are the files you have deleted from your computer. Files don’t get corrupted or deleted on the storage server. Some files stay in your account [on the SugarSync server] even after deletion. You can delete them safely. Your data will not be affected in any way. [That's a direct quote]

My interpretation:

I can only guess at how these orphans appear, since deleted files actually appear in the “deleted files” folder.

Idea 1: SugarSync maintains up to 4 previous versions of the file you synchronise so that you can roll back to a previous version. When you delete files they are moved to the “deleted items” folder, but previous versions could be orphaned and appear in the “reclaim storage”.

Idea 2: Because I had to reinstall the software, create a new link between the laptop and the Clound, and delete the old link via the online control panel, the resulting rescan picks up some of the old associations but also creates new ones. These new ones cause the old data to become orphaned.

Reality:

SugarSync doesn’t work properly. I had to re-install SugarSync on my Vista machine a few times (this is the Sharpcast support team’s answer to most of your problems), and when you do this you have to link your computer to the SugarSync account again and delete the previous computer relationship online. This then causes SugarSync to re-scan the machine and it often uploads/downloads some files again. It also fills up your desktop recycle bin with old copies of hundreds of updated files if you have a large data set.

It would be nice to be able to trust that SugarSync is doing its job properly (as Sharpcast asks you to do), but after two weeks of not getting a straight answer to the problem and being told to (yet again) re-install the software, I took fate into my own hands and discovered that you can selectively delete or archive files from the “reclaim storage” list of files. I chose to delete 2 files I knew I could find easily in my 5Gb of data and deleted them permanently. I waited a few hours and checked to see that the original files were still safe – as SugarSync claimed they would be.

Unfortunately, one of the files – a client proposal – had been deleted. Since only 1 file was affected I could easily recover it from the recycle bin on my desktop. However, the “reclaim files” dialogue lists thousands of files and folders in my case, and given that SugarSync has just been proven to delete files that it shouldn’t, I have now stopped SugarSync synchronising any data between my computers for fear of loss of any further data.

This problem may not be a direct consequence of SugarSync, and may be due to my Vista Home installation on the laptop potentially conflicting with the XP Pro installation on my base computer. Of course I am looking at all angles here – but the bottom line is that SugarSync doesn’t work in my operating environment.

Other Solutions

These all work in much the same way as SugarSync – in the sense of you create an online account and install the desktop client software, so I won’t cover this part of the process again here.

DropBox

My initial reaction was that the team at DropBox are much more open with their product. Sharpcast are clearly in the “old school” in that the product is a black box and you just have to take it on trust that it works. Their blog is essentially release notes rather than communication, the latter approach being better aligned with a product embracing cloud-based computing IMHO.

I was pleased by the DropBox documentation and the explanation of what it’s actually doing in order to synchronise your data, unlike SugarSync which doesn’t even appear to have a user manual. DropBox also has active user forums where you can get fast responses to issues from other users.

DropBox also has a nice feature where you can share sub-folders with other people and so collaborate on files over the web in true virtual teams. SugarSync is a closed system for you alone, although you can “send” files to people but once you do they are outside of the synchronisation control loop (this is basically an email tool).

The downside is that DropBox currently only works from within a single root folder called “My DropBox” which is created under your “My Documents” folder. All files which you require to be synchronised must be located in this folder. I have about 30Gb of data that I would like to keep synchronised and don’t really want to move the files. DropBox do plan to extend this in the future to allow you to use your pre-existing folders but it’s not available at the time of writing.

MobileMe

I had a quick look at MobileMe from Apple. However, file synchronisation doesn’t appear to be built in at present (according to the website) and the system only provides an iDisk feature where you can drop/upload files to create an online copy. However, I am told by a customer who uses it that it does support full synchronisation and that they use it regularly (see * in table below). Also, the account is limited to 20Gb which is not sufficient space for my files .

The email, calendar and PIM-related synchronisation are excellent, but since I already have these features through Emissary Mail this would be duplication of effort and potentially lead to confusion!

Live Mesh

I then looked at Live Mesh from Microsoft. This is currently a tech preview (alpha) release and only offers a 5Gb free version at present. However, the interface is much easier to understand and the development blog is much more open that Sharpcast’s. There is even a post on tombstone files – which sounds very much like the problem I could have been experiencing with SugarSync.

Live Mesh is Windows-only at the moment, but Mac and mobile versions are on their way. It’s early days, but as a Windows user I hope the operating system integration is going to make the software more reliable in synchronising my files. Live Mesh also allows collaboration via public folders which will help working with my PA and other team members. Plus it has a built in remote desktop feature not present in other replication tools which gives you full access to other computers fully connected to the mesh.

From a perspective of vision, Live Mesh appears the most capable product long-term. It embodies both the synchronisation and replication features with cloud storage plus principles of mesh computing by allowing you to connect to the peer computers in the mesh itself, so extending the reach beyond the cloud. The current level of issue management (i.e. what to do with file synchronisation conflicts) is much more transparent and it works!

When a file conflict occurs – for example when you edit a file on multiple machines – Live Mesh shows a dialogue that allows you to choose which file to keep, or you can view the files and decide what to do.

The only issue I have with Live Mesh at the moment is that it summarises what it has done – e.g. “Edward Terry added 16 files in the folder Mesh Quickshare” – but there is no way to find out which 16 files. When you are dealing with large amounts of data, it would be good to be able to access the details to verify manually that everything is as it should be. Live Mesh has also (on two occasions) appeared to have performed a large update of files and folders even though nothing was actually changed in the folders it identified. I am not sure if this is just the desktop catching up with the Cloud, or something to do with the local computer time changing, but it was a bit confusing. All the files appear to be present and accounted for, so I am hoping this is just the “news” feature of Silverlight catching up with reality.

Function Comparison

In conclusion

It’s still early days for Cloud-based systems and the file deletion by SugarSync is a serious blow to the credibility/stability of an application you are basically trusting your business to.

I can’t speak for the reliability of DropBox or MobileMe as I haven’t tried them.

Live Mesh is still only a “tech preview”, but appears more reliable even at this early stage. The online reporting explains what is happening more clearly and Microsoft’s recent fix for tombstones is very timely. It would benefit from a little more detail in the audit trail as “20 folders added” is a little brief – a link to show you which ones would be a nice addition.

My initial experience with cloud storage is positive overall and I will continue to pursue this route. Once the products mature and the minor issues I have experienced are resolved through better support and refined interfaces, these technologies will help more people be truly mobile and the idea of the ubiquitous Internet will help transform how we do business together.

Go Further

• What will Happen to the Web if Cloud Computing Really Succeeds?
• SugarSync
• MobileMe
• DropBox
• LiveMesh
• A five-part Backup Strategy