My Azure Hosting Hiccups, or "How to Shoot One's Self in Your Own Foot"
As you might have read, I moved my website onto Azure a couple of weeks ago. I have not looked back at all. Well, okay. Two events made me rethink my strategy around hosting on Azure. One was my own doing and the other is a conflict between DotNetNuke and the Azure SQL model. Both were resolved and I am again 100% on hosting via Azure, until the next problem rears its ugly head.
Let's review how I got to today. First, I started out on Azure with my DotNetNuke instance using the DotNetNuke Azure Accelerator. It was a miserable failure and I was floundering. I also had other issues going on that night with various technologies and decided to skip it. Then, I found the ease of setting up my Azure hosted DotNetNuke CMS system. Success!
Let's move on to last Saturday, March 2nd. I decided to do some re-configuring of the website on Azure. First thing, I reviewed my account and my bandwidth and processing needs were pushing the limits of the free account. I had to change from their "Free" webhosting instance to the "Shared" model. On top of that change, I wanted the URL to be my own website's URL and not the azurewebsites.net version that is created when you setup a website on Azure. Lastly, I wanted to use a publishing system so I could upload changes to my site when update came out. In my case, the only one I had some experience in (and not very much as I find out) was GIT but I did not want to tie my Azure site to GitHub, so I selected localized GIT on my desktop. With all of these actions, I pulled out the gun, filled and loaded the magazine, chambered a round, and pointed it at my foot.
Sunday morning rolls around and I get a text message page at 6:30 am; my Azure website is offline. HUH? How can it be offline? Did Azure have another one of their illustrious outages? Looking at the site on my phone, I got at 502 error. Ummmm … "Bad Gateway"??? Thinking my DNS was having issues, I went to the default Azure website URL and got slapped with another 502 error. My site was down! Jumping out of bed, I fumble into my computer and start to look at the issue. I pulled up the Azure Portal, my site, my monitoring services and my VM hosted mail server to get an external perspective on the issue. No matter how many times I pressed SHIFT-F5, the site was down. I checked all browsers and still the same. I had the monitoring service check from all of its servers; still down. Looking through the Azure portal, nothing seemed to be misconfigured. Checking the Azure DB, no issues were seen there. Last check was looking at the webserver logs from Azure; the logs did not show anyone visiting the site. Huh? How could my attempts from my phone, home computer and hosted VM not register in the Logs. I restarted the website services and nothing in the logs. One more SHIFT-F5 and "Ta da!", website functional. HUH? BLAM! That hurt.
I don't like having mysteries. One of the toughest thing for me in my IT world is to have something fix itself and not know what the root cause is. Many of you might remember IBM's commercials around the "Self-Healing Server Pixie Dust". I mock these commercials because parts of servers can fix themselves but others cannot. System Admins are still a necessary group of people no matter what technologies you add to hardware or software. Giving those professionals the 
information they need to perform good root cause analysis is more important than self-healing. Yet, this is what I was looking at. Nothing in the logs, in the stats, nor in the code told me what was wrong. Nothing like this happened the 7 days I was hosting it on the "Free" model. Being a good IT Operations person, I started rolling back my changes. Doing the easy stuff first, I reversed the DNS work and then went to breakfast. During my meal, I got 10 pages that my site was up, then down, then up, then … well, you get the idea. After breakfast, I went home and switched the site back to the "Free" model. I waited for any changes and was met with similar pages and watching my site go from non-responsive to responsive. My final thought was that the problem must be in the GIT deployment system.
The story turns very interesting at this point. Reviewing the settings for Azure, there is no way for an Azure administrator to remove a deployment system from a website. No mechanism is in the Azure Portal to change once a deployment system is selected. I was stuck with an unstable site and no way to revert back what I did. It seems Azure's method is to just recreate the site. I copied the code from my Azure website to my local computer, deleted the Azure website and created a new one in Azure, copying the code back from my desktop. Thanks to many factors, the file copying seemed to take hours though, in reality, it took 35 minutes for both down and up loads. I clicked on the link for the new site and ".NET ERROR". A huge sigh and facepalm later, I delved into what was going on. DotNetNuke was missing key files; my copy from the internet did not include them. Instead of trying to figure out where I went wrong, I reviewed what I had: an Azure website with code that was bad and an Azure SQL DB with my data. To make it easy for me, I decided to just build a new DotNetNuke installation from scratch with a new DB. Then, recopy my blog data back in to complete my work. After approximately 2 hours of work later, my site was back up and running again on the Azure URL. Success!
Going over all of the changes I wanted to make, I decided to separate out the changes and leave them for 24 hours to verify that it would not affect my site. The critical change I needed to make was changing from the "Free" mode to the "Shared" mode for the website. Azure would block the site if I did not do this because I was over my resources. This was a "no brainer" for me so this was my first change. I re-enabled my redirect from the server that hosted this site before and all was working again. Monday night rolls around and all has been stable. My next change, the URL to my domain name, was prepped and executed. My site was stable for the rest of the night and into the next day. My analysis was correct, the configuration of GIT as a "publishing" system was the cause of my outages on Sunday. Tuesday night led to a lot of review of Azure web publishing. All of the information I was able to find led me to my final conclusion; I am not developing my own code and do not need publishing. None of the systems would help me and only looked to make things more difficult. In its current mode, I can FTP files up and down from the site which is good enough for me.
Let's move on to Wednesday. I received a notice from DotNetNuke that they released 7.0.4 of their system and my site is currently running 7.0.3. I should upgrade it to make sure I am safe, secure and stable, right? As I started to download the code for the update, I got the gun back out again, filled and loaded that magazine, chambered a round, and got it aimed right next to the hole I put through my foot on Sunday. Using FTP, I uploaded the update code and pulled up the upgrade installation page. I waited for the upgrade to complete while working through my e-mail. When it completed, I turned and saw "Completed with errors". BLAM! I got to stop shooting myself like this.
One of the modern advantages of DotNetNuke is the logging that upgrades and installs do now. I was able to pull up the installation log and get the exact error messages from the upgrade installation: 3 SQL errors when it was processing the SQL upgrade statements. Looking at each error, the error messages were confusing to me. In two of the errors, the upgrade tried to determine if an index was in place and then remove said index to replace with a new one. Yet, when this was performed on my Azure DB, it threw an error saying "DROP INDEX with two-part name is not supported in this version of SQL Server". How am I going to fix this? For those of you that don't know, my start in IT was in SQL DBA and programming. I dug out my rusty SQL skills and started through the database alongside online the MSDN website for Azure SQL. In no time, I figure out what I need to do to modify the DotNetNuke code and run the SQL statements against my Azure SQL DB. The third error was even more interesting. The DotNetNuke code wanted to verify that a default value was set for a column in one of the tables. The way this is done normally in SQL Server is to query against the sys.sysconstraints system view. The problem with this in Azure SQL DB is that there is no sysconstraints view available. The SQL statement that ran returns "Invalid object name 'sysconstraints'". More digging and I found my answer; Azure SQL has the new Catalog Views of check_constraints, default_constraints, and key_constraints available. Quick change to using the default_constraints view and I found that the desired default was in place. My upgrade is now complete and a success.
As you can see, I did all of the damage myself; I cannot blame Azure for it. My impatience to not read all the way through and just get things going caused my own downtimes. I have no doubt my thrifty behavior will also be my downfall when Azure has any sort of outage in the US West Websites or SQL DB layers. If I want a website that will not go down, I need to create and pay for the Azure infrastructure to do that. For now, I am super happy with my decision. To the cloud!
Are you thinking about moving your website into a cloud provider? If not, what is stopping you from doing that? Post your questions and comments below.
What Does Your DR Look Like? Or "Holy #$*%! Everything is down!"
This is a topic near and dear to me these days. Having suffered a recent outage at my job with over 9 hours of downtime, this is now a major issue for me to work through. Everyone always gives disaster recovery (DR) lip service. They come up with ways to backup data, provide alternative networking access as they can afford, and try to create plans. My feeling, much like what I have dealt with at prior positions, is that no one really invests into DR. I hope to provide a few cautionary tales to help you convince your management to make the investment.
Disaster recovery is insurance. All the investment that is made in DR is insurance against a downtime. At the same time, everyone keeps saying "it will never happen to me." I can provide references that it does happen and the outcomes can be brutal for a business. Downtime can lead to loss of business opportunity, change in customer perception reducing their business with you, loss of customers entirely, or complete collapse and closure of the business. To offset these outcomes, businesses invest in disaster recovery to mitigate the impact of downtimes.
When looking at DR, first thing that people need to determine is what are those critical systems; what systems do you have that if you lost them would impact the business most. For a manufacturing company, it could be their control systems for their machinery. For a datacenter, it could be the power and networking systems to keep the hosted systems online. For a healthcare company, it could be all the systems involved with patient care. The IT team needs to sit with the business and management teams to determine which systems are those critical systems and all of the infrastructure that supports it.
Now that the critical systems are identified and their infrastructure is determined, a full risk assessment of those systems and infrastructure needs to be completed. Are there devices that have single points of failure? Can servers be connected to the network in diverse paths, also known as teaming? Can the software be setup in clustering technologies to allow more than one server to be setup and kept in sync? What equipment is the oldest and have a higher possibility of failure? Working through the risk assessment with knowledgeable team members in both the IT and business teams will help find the answers quickly.
Now that the risks are identified, the professionals need to step in and make some plans to mitigate those risks. That planning can include duplicate systems, cluster creation, backup and recovery techniques, additional networking equipment and lines, and warm/cold spare hardware to name a few. Each of these plans need to be fully thought out including the costs of creation and ongoing maintenance.
Part of the maintenance of backup systems is using them, a largely overlooked step of DR planning. Both business and IT teams need to role-play disasters to ensure these policies, procedures, and systems will work. These sorts of tests interrupt normal business operations but should be done on a regular basis to ensure all systems are go for a real disaster. After each test, the affected teams should get together and review the test event to improve policies, procedures, or systems in the future.
I know that what I have said so far is something that everyone else has said to their management to push for better DR planning and testing. I have said it myself at times. Having gone through a large outage that affected my company's business has brought it to the forefront for me and gotten the attention of my company, a company that runs 24x7 for our business. We lost our primary datacenter, the hosting location for primary servers and the hub of our network, for approximately 9 hours on a Thursday night, which is our busiest times of the week. While we had some basic processes and procedures in place, it was thanks to the hard working teams at my company that we made it through the outage.
During the outage, the primary datacenter lost its primary power at the Automatic Transfer Switch (ATS) that allowed them to select either the utility company or their generators as the power source. Not only did they lose the power there, the ATS literally blew up blowing out part of the wall behind it. In trying to get the datacenter power back online, they also found that a fuse in the transformer was bad, possibly causing the whole problem. To correct the transformer fuse, they would have to fail their second power source from the utility to generator to allow the utility to pull a fuse from that second transformer as the utility crew did not have a spare on hand instead of waiting up to 2.5 hours for them to go get one at their warehouse and returning.
While seeming a simple fix, this would have impacted part of the datacenter that was still operational and hosting one of their biggest customers. That customer did not want any more change introduced into their hosting systems. As a customer impacted by the continued outage, I pushed on the datacenter to start the change with haste. This put the datacenter in the middle between customers.
Eventually, this was resolved and the generator added to the second circuit, allowing the utility to repair the primary circuit. This is where good process and planning helped out my team because we knew which systems had to be started first and what order to effectively restart our business. Once we got our systems up, the business teams started in cleaning up their issues from the outage.
After the outage, an emphasis was placed on all parts of my company to determine ways to improve our business resilience to outages. This includes alternative network connectivity for outages, secondary datacenters, hardened systems, and improved policies and procedures to reduce the impact on our customers if we have another outage.
I will admit that I wrote this blog entry a while ago but could not finish it off until now. It was difficult to read what I wrote because it would make me go back and remember all that happened; reading my blog entry brought back all of those memories and feelings as if they were happening again. Major service interruptions are difficult for any group. What made this worse for me was that there was nothing I could do but wait for our hosting provider to fix their facility and services. Since this occurred, they also have taken some steps to improve their offering to ensure clients like my company do not suffer through something like this again. Improvement can happen for you directly or for your providers and partners.
The key takeaway is that outages will occur. The better your systems and networks are designed and the more time is invested in both business and IT policies and procedures, downtime impact can be reduced and customers can be kept happy during those outages. The best outcome that IT and business teams can hope for is no impacts for their customers at all while systems are offline or unavailable. No single system can stay 100% available forever but well-designed systems and networks can offer the "Five 9's of availability" (99.999%) or no more than just over 5 minutes per year of downtime.
What are you doing for your disaster recovery? Is it even a thought for you or your company?
Moved my Site and Blog to Azure … How Easy!
Well, I finally made the switch. As many of you can see in the URL, my blog has moved from my personal servers onto the Azure fabric. It is something I wanted to do for a while and never got quite around to finishing until now. It is not totally done but I am happy with the interim results.
For those that don't know, Azure can offer easy web hosting up in their cloud with CMS systems like WordPress and DotNetNuke. I personally do use DotNetNuke and have for several years. Installation was looking to be interesting thanks to a few projects around like the DotNetNuke Azure Accelerator. Other blog entries and wikis are out there talking about how to get this accomplished.
A few weeks ago, I tried to use these "recipes" and failed miserably. In the same evening, I also screwed up local installs of some test servers and thought if I could just strike out at a bar, the evening would be complete. The process seemed to be fraught with missing settings, steps that did not work as advertised and some complications, later found out to be caused by Azure issues.
The next day, I sat back down and looked at the Azure offering and within the "creation" workflow for a website is a "From Gallery" option. For grins, I clicked on it and the world got so easy for me! Within this option, Azure offers a multitude of predefined systems for installation from their Azure Store. This includes many CMS systems like DotNetNuke, Drupal, Joomla!, WordPress, MediaWiki, Orchard and many more. It also includes E-commerce engines, forum systems, galleries, and wikis. Right at the top of the list is DotNetNuke Community Edition, my choice for CMS. They make the Professional version available as well but that is a paid product and I know what I am doing with the CMS engine.I started down their wizard path to creating my new Azure website using the Gallery image of DotNetNuke Community edition 7.0.3. Clicking the "next" arrow brought me to their initial configuration screen where I put in my Azure URL, told it to create me a new DB for the project, and allowed me to choose the region for hosting, West US in my case. One more screen for the DB setup on a new server, the DB username and password, and which region for the DB hosting, West US again for me, and we are off to the races. The next steps are very DNN specific so I will not bore the majority of my readers with those details.
Once all was setup, I could browse to the Azure Base Site URL and look at my new DNN installation. Within 10 minutes, I had my beloved DotNetNuke 7.0.3 running in the Azure cloud without any major work on my part. I was able to install my favorite blogging module, Live Blog from Mandeeps, and thanks to my SQL knowledge, port over this blog from my personal server to the Azure site. A quick set of redirections and here you are with my new version of my blog. Now, I just need to get more content up here…
Have you started using Azure for hosting of your sites? If not, why not give it a try with a 90-day trial? Sign up at http://www.windowsazure.com/en-us/pricing/free-trial/
Tips to Cut the Cord With
"Cutting the Cord" is a catch phrase that is thrown around in this modern age. The main meaning is ability for many people to remove services that they used to pay for that seems redundant in these changing days, primary being television and telephone services. For me, I am fully cord cut when it comes to telephone and nearly cut with television. I will explain what I have done, how I came about my decisions and what it took to execute my cord cutting. In the end, there is no way to cut all cords unless you want to disconnect from the world and entertainment. Instead, the goal for most cord cutters is to run all of their needs across their data service lines. What you need to do is find your goals for cord cutting and then find what will help you achieve those goals.
The easiest service for me to cut was the telephone. I live on my cell phone; I know many others that do as well. When I used to have a phone line, I was paying nearly $50 for something I rarely used. My concerns for dropping landline service keyed around people getting a hold of me and emergency services. Since everyone had my cell phone that I wanted to contact me, that first concern was null. As for emergency services, the concern is proper location services for emergency crews to arrive at. Cell phones are now required to have E911 (Enhanced 911) location services but this is not a guarantee. Instead, I use a little known fact.
My condo already was wired for phones and that service is attached to the local phone carrier. I can plug a phone into that line and call 911 without costing me anything. This is perfect for emergencies and the 911 operator will have the location information for the line that was established by the phone company. For everything else, I use my cell phone.
My solution for phone service was easy for me, but it won't be easy for everyone. There are some good alternatives out there including Vonage and Ring Central that provide VoIP solutions over your broadband data connection to Skype and Google Voice that provide some call management and VoIP features as well. Think through what you need for yourself and your family. Then, find the service that provides what you are looking for.
Now that I have started cutting the cords, I reviewed my television entertainment needs. This will vary from person to person, not just as a whole family unit. Those needs can change over the years which will mean that flexibility is key. Let's use myself as a test case now. In 2004, I used to be a background TV person where I left the TV on all the time not really noticing what was on. Over the years, I have changed my consumption habit to enjoy specific programs. These changes were both caused and caused by my cord cutting choice.
I find that most of the shows I like to watch are available on the main networks, or specific shows on cable channels. Since I cannot purchase an a la carte cable package and do not want to pay up to $75/mo. for the small number of channels I want, I worked through the shows legally online. I start with channels that my local cable company offers via their "Basic" package (Comcast offers at $15/mo). This includes the local network stations (ABC, NBC, CBS, PBS, Fox, WB, CBC, Ion) in HD, Discovery Channel in SD, and several local off-band stations. It also includes stations that I do not care for (religious, 24 shopping, government access, non-english) thus making it a normal cable subscription. Depending on your state regulations, not every cable provider offers this basic subscription. You should be able to get these channels with a TV tuner (in a TV or in a computer) that can get digital cable in the clear (Clear QAM). However, some companies still require one of their set-top boxes to get even this "basic" package. I will be cutting this service if that occurs with Comcast.
Add to the basic cable subscription, I utilize several online services to watch episodes to fill in on channels I do not have. With my desired list of shows, most of them are available online via services like Netflix ($8/mo.) and Hulu/Hulu Plus (Free on web/$8/mo.). When I add up all the entertainment I get that way, about 90% of the TV shows I want to watch are available. To fill that last gap, I utilize Xbox Video (formerly Zune Video) and Amazon Instant Streaming. One or both of these services has the rest of the shows I want to see available to purchase with pricing based on the length of the season and the quality. There are other sources like iTunes but I choose not to use them as my devices are not well matched for it. With smaller cable subscriptions and online sources, you can find most of your content that you want to watch without paying the high rates of cable. Want even better news? You now have more options to watch entertainment thanks to the internet.
Remember that one of the changes I have undergone is having the TV on in the background to now watching specific shows and paying attention to them? This change alone reduced the amount that I was watching and helped to filter what I watched to a very specific set of shows. It also added a brand new source of content that most forget is available, the internet. Content creators have started to understand they do not need to work through "traditional" media publishing channels. They can create a website and an RSS feed to launch a "video netcast or podcast". Some large media people have jumped over to this new medium such as Leo Laporte with his TWiT network, Adam Carolla with his Adam Carolla Entertainment network, and former MTV VJ Adam Curry with his Mevio network. Others have been on the internet from inception like Audible for audiobooks and special interest sites such as Technet on Microsoft for Microsoft IT professionals. In my experiences, I find this content better than the content coming from the networks and cable, making me miss the deluge of cable channels filled with programs I never watched.
Now that you have done some homework on what you want for content, you need to think about how you are going to consume that content. Since I started with the notion of replacing television, I will focus on the use of a television in either a living or bed room. The easiest devices are some sort of set-top box that has the content available through apps. I have a Roku device in my bedroom and installed a Windows Media Center PC in my living room. I am not the norm though here in that I built and managed a computer that was a DVR/set-top box. It was the most flexible and offered all the content but not all in a 10-foot UI. Some of the content I had to use a web browser with a wireless mouse and keyboard to access. I was willing to go through that while others are not.
To make things very simple for the average user, you should really look at the Roku devices to plug into online services with their applications. They offer applications that connect to content services like Netflix, Amazon, Crackle and Hulu Plus. In addition to the major content services, Roku devices can connect to many new media companies with apps like TWiT, Revision3, The Onion News, and CNET. This gives the Roku devices a big advantage in the fight for a single box to add these services. Even some traditional television channels have applications of their own on Roku devices like CNBC, Sail TV, Fox News and NBC News that offer live feeds from their cable channels online. HBO even has their HBO-to-go service, targeted at mobile devices like tablets and phones, available on Roku devices. For this to work, you need to have a cable subscription with HBO added to it to access it. The Roku devices could be easy, cheaper set-top boxes for additional TV's in a house to reduce the need for cable/satellite set-top boxes.
Recently, I changed my living room to use a Xbox 360 as the primary device. With apps for Hulu Plus, Netflix, Amazon, YouTube and other online services, connectivity to my Media Center for recorded and live content, native support for Xbox Video, and it's DVD drive, it is a single device that I can use. As my media collection does not include Blu-rays, the HD content I get is from online sources so I am not hurt by the missing Blu-ray drive of the Xbox 360. If I paid for a full cable subscription, I could use my Xbox 360 as a set-top box for their services on Xfinity and FiOS. Through the Xbox devices, I see Microsoft trying to make a play for the living room via easy to use devices and I get that now. Rumors are with the next release of Xboxes coming in 2013, we might see a specialized media only device along with a new gaming unit.
As you have seen, I have done a lot of research on what is best for me given my consumption of entertainment. What works for me may not work for everyone. One key demographic I can see is families with children. The story here is improving with a "children's focused" Netflix integration and view along with specific apps on Roku for kid's programming. Parents need to research what is best as there is so much content available on the internet. While it can be overwhelming, it is the same thing I would expect most parents to do with other forms of entertainment. Most of the better systems to allow for parental controls to manage what kids watch but it does not beat being there and watching with them to know what they watch.
Cord cutting is possible today, even though we are in the early days of it. You see large media companies trying to slow or stop it as much as they can to keep their current revenue models flowing the dollars to them. Accepting that some content will not be available for a long time or ever is one thing a "cord cutter" using legal sources has to accept. Just one example of this for me is Game of Thrones from HBO. Without an HBO subscription, requiring a much higher cable package than I had nor wanted, I accepted that I would not get it until it was released on Xbox Video or Amazon, nearly 1-2 years later. Spend the time figuring out what you can live with and without, where can you source it, and what device can show it on your preferred screen.
The last thing I will mention is most of this requires a broadband style network connection and can push usage caps if they are attached . Since I am an IT Professional doing a lot of work online and knew I would be using media services, I purchased business class internet that provides to me 25 Mb download guaranteed with no caps. This is not cheap internet at $110/mo. and if added to everything else, might push someone back to regular cable service. I use the bandwidth for more than entertainment so I feel the monthly costs for that inexpensive and would rather put my money for that over a television subscription. Add it all up for yourself and figure out what works best for you.
What sort of cord cutting have you done? What are your goals with cord cutting? Let us all know through the comments below and help others get out the wire cutters.
The Good, The Bad, The Ugly … The Wonderful World of Compliance in IT
"Compliance" is often perceived as such a dirty word to IT professionals that it might as well be censored. The mere mention of "compliance" brings about visions of additional paperwork and processes that slow down everyday tasks and project schedules for many IT pros. With newer regulations, be them federal laws such as Sarbanes-Oxley Act of 2002 (SOX or SOX404)[1] or Health Information Portability and Accountability Act of 1996 (HIPAA) [2]; association or vendor regulations like Payment Card Industry Data Security Standard (PCI DSS) [3]; or internal standards created by management, IT teams in both engineering and operations have to work to meet these regulations and standards as a part of their project and daily work. This was a great discussion topic for Denny Cherry and me on his People Talking Tech Podcast. [4]
Want to make an IT team squirm? Create a meeting about "compliance" or introduce an auditor or consultant. Let’s be honest; we’re technical people, and we want to make things work as quickly and efficiently as possible. It’s uncomfortable to have someone looking over your shoulder to verify that you are “doing things right," either through the operations team installing and configuring, or developers writing code that is deployed to users for saving data into the server or the cloud. But get this: it is actually in our best interest to see it from a different angle. Compliance, standards and regulations are the IT professional's friend. Much like all other aspects of IT, with proper planning and execution, complying with standards and regulations ensures that you have "air cover" for everything you do.
Proper planning for compliance is just like any other IT project: the earlier it can be integrated in plans, the easier the execution can occur. This is true with engineered projects by developers or with integrated projects by operation teams. Compliance can range from smaller tasks such as documenting what is built or installed, all the way to deep logging and intricate permissions management systems. In most situations, there is no "silver bullet" solution to comply with regulations or standards, and any vendor offering this to you or your company should be reviewed carefully. These sorts of vendors typically try to engage with non-technical management to sell them on solutions that the IT team has to later "figure out how to integrate them." (If you have good horror stories around this exact situations, feel free to share in the comments below.) To get ahead of those "snake oil salesmen", be ready to show your management how you are currently or will meet your standards and regulations.
Identify the Requirements
First step in creating a good compliance plan is to understand what you need to comply with. This can be fairly straightforward for some, such HIPAA for healthcare, while being much more complicated for others, combining SOX with PCI DSS or US state and federal regulations with other countries. This step is very critical and will require IT professionals to reach out to the business users they support and possibly consultants like lawyers or compliance officers. It may sound simple, but this can be the most difficult step as many regulations are not black-and-white. Each person can read the same words and interpret it differently. Documenting this interpretation as it is being reviewed will only help you in the future if you have to defend that interpretation from regulators or re-interpretation with new staff or consultants.
Create and Socialize the Plan
Second step is creating internal processes, procedures, and standards that meet all of the items you found in the discovery. Many companies have unwritten ways to do things, rules that everyone follows without question, and systems they use to track what is done and how they do it. While some companies do a great job in documenting their processes, procedures and standards, most do not , and getting teams to change this practice can mean a cultural shift.
Where you often see this issue is when smaller companies grow larger. Small companies consider their IT teams agile because they are all generalists, and any member can and does fill all possible roles. As companies and their IT teams grow, specializations occur. The company's business users yearn for those old days when they could call one of the IT team members to get help and the IT team just got it done. The fight to follow processes and procedures while "getting things done" is a constant struggle. Selling the benefits of following processes and procedures internally is one of the toughest things for IT management to do in situations like this.
Document and Organize
The third step is to take all of the documentation and organize it so that it can be reviewed, executed and tracked. This includes processes, procedures and standards, along with how the team is executing against those standards. In many cases, regulators can show up without notice and audit the company's compliance. Without having the information in an easy to access system or storage, it is useless to both the IT team and the auditors. Again, there is no "sliver bullet" solution for this. Every team needs to find their own solution that works for them. For some teams, it could simply be a file share with Word and Excel documents; for others, it might take a self-developed or commercially available software package.
All of this seems easy on paper, but there is no quick solution or answers. IT teams, management and business users need to take their time to understand what needs to be done, and when they need to meet regulations and standards. Sometimes, one group may ask for more than what is required in the regulations. This needs to be tempered with timelines and costs. Lastly, remember that it takes time for people to adopt changes, and anticipate that when creating the project execution plan. By working together with realistic timelines and good communication, a proper compliance plan can be executed.
Plan for Continuous Improvement
Once the compliance plan is pulled together and the users and IT team are following the plan, the best thing to show most regulators and auditors is a continual improvement processes. Regulators, auditors and compliance officers love to see improvement. In some ways, it is better to create your initial plan and then slowly improve that plan over time rather than trying to create the "perfect plan." Improvement around compliance is seen as a good sign of compliance in an organization. This makes the improvement process just as important as the initial compliance.
IT compliance does not need to be a dirty word. Everyone has his or her stories around the good, the bad and the ugly of compliance; the stories of well executed plans, stories of bad or no plans, stories of regulators imposing large fines and sanctions. Take your time to prepare and execute the best compliance plan you can with the resources available. Once that plan is in place, create an environment that makes ongoing improvements as painless as possible, so that compliance is something everyone understands and wants, and not seen as an impediment to their work.
How do you feel about IT compliance? Do you have stories to share, whether good or bad? If so, put them into the comments below.
This was cross-posted by Veronica Wei Sopher on the Born to Learn Blog at MS Learning. You can check this one out specificially at http://borntolearn.mslearn.net/btl/b/weblog/archive/2013/01/28/the-wonderful-world-of-compliance-in-it.aspx and other great posts over at http://borntolearn.mslearn.net/. Special thanks to Veronica for helping with this posting.
Notes:
1 - More information about Sarbanes-Oxley Act of 2002 (SOX or SOX404)
2 - More information about Health Information Portability and Accountability Act of 1996
- http://www.hhs.gov/ocr/privacy/hipaa/understanding/index.html
- http://www.cms.gov/Regulations-and-Guidance/HIPAA-Administrative-Simplification/HIPAAGenInfo/index.html
3 - More information about Payment Card Industry Data Security Standard
4 - Direct link to my appearance on People Talking Tech, January 22nd, 2013