Some customers have inquired about our development, testing, and release process. Although it may appear at times chaotic, there is actually a method to the madness.
We formally define a development "iteration" as 2 weeks. November 2008 marks the occurrence of "iteration 200", which is a pretty exciting milestone for us (the first release was in March 2001). We do updates to production sites every 2-3 iterations (4-6 weeks).
At the beginning of each iteration, development tasks are documented and prioritized based on a number of factors that include:
a) customer feedback
b) exception/performance report analysis
c) industry trends
Our Premier Support customers are provided with their own workspace at
http://www.centraldesktop.com/. Requests posted to this repository are typically given highest priority.
Next, a work breakdown structure is created for each task and delegated to the appropriate resource for estimation and development. If a development task will take more than 2 weeks, a particular module is said to be "under refactoring" and the changes are spread out over several iterations.
A formal design process typically follows that may involve a quick whiteboard discussion or a detailed Visio diagram (I've started using
http://www.gliffy.com/ to eat our own SaaS dog food
).
For more complicated features, a "test first" approach is taken and unit tests are established to assert expected outcomes. These unit tests are also maintained and refactored along with the core product for use in regression testing.
Build files are updated and releases are internally tested using a regression test suite (see screenshot below). These tests ensure new changes have not broken existing features and also ensure 100% backwards compatibility with previous releases.
Occasionally, some releases require programmatic migration in order for new features to work. This rarely occurs because of our use of an object oriented metabase instead of a traditional database, but when it does, there is a corresponding framework for including migration batch files in the release. Lately, we've been empowering customers to initiate these migrations manually, such as the recent ability to convert a plain HTML editor to a Dialogue Script editor.
Release candidates are slowly deployed to our
own website, the
Developer's sandbox, and other staging environments for some real-world acceptance testing.
Operations is then given the green light to proceed with updating all production environments, at which time scheduled downtime notices are sent to customers. We typically do not upgrade more than 10 portals at a single time, preferring to rollout changes over a few days. Because of the mission critical nature of many clients' websites/portals, we often let customers determine preferred schedule maintenance downtimes.
An upgrade only takes about 30 seconds to apply. Total downtime is often no more than 3-5 minutes, but we may use the allocated maintenance window to optimize databases or make other changes that may impact the site during scheduled maintenance.
A number of monitoring services are utilized and watched closely after an update and the operations team has the ability to rollback a release to any previous version at any time.
We don't always get this process 100% right. About 5% of the time, a formal release may be followed the next day with a minor patch release to correct any missed gaps. I would like to see more test coverage in our regression unit tests, but this is not a silver bullet. The increased use of Javascript in rich Internet applications is much harder to test and new tools are being developed for this purpose today. Because our core feature set offers a programming language environment, it is difficult to know how changes may impact some customer developed applications (more on the pros/cons of requiring customers to write their own tests in the future).
I hope this post sheds some light on our process. The great thing about the SaaS subscription model is that you receive all these services in the monthly subscription and we ensure every upgrade is done to perfection, which results in happy customers.
There are definite benefits to using cloud-based services for business users, because it relieves them from worrying about the technical details of maintaining their website or portal. But many IT professionals who have years of experience managing this same process also appreciate the benefits of outsourcing the maintenance of their website to the cloud so they can focus on adding business value to their organization.
