Howe is your power infrastructure laid out

It’s time to rethink the way we do power in our offices  and homes.  Time and time again we see how business and private homes is hit by flooding.  And what is the problem.  All electricity start at ground level or below.  Due to the unsigthliness of overhead power lines all such feeds are now buried under ground.  Then just inside there is a power meter and then the main fuseboard, And circuits spread out from there.   But with flooding this is a very vulnerable layout.  Even a foot of water can be an issue if these things are in a cellar, and electricity and water makes a lethal combination as the recent floods on the East Coast of the US proofs.  

In Ireland all houses have to have a water tank installed in the attic to ensure a certain water pressure, and for a certain amount of emergency supply if mains pressure fails.  Your power needs to be arranged in a similar manner.  The main lead needs to be brought fully insulated up to the top floor and the main breakers placed there.  Then each floor need a main switch so they can each be isolated.   That way you can still have safe power even if the ground floor is flooded.  Just switch off  that floor.   This panel needs to be in a place high up tough.  You can’t stand in a foot of water switching electricity.   For extra security you can also have automatic sensors that can switch off each floor as required.  Even better combine it with a complete environment monitoring solution.  The hardware for a solution that measures temperature and have sensors for power outages and flooding and notifies you remotely, can be bought from as little as a few hundred.

If you have a ups and or generator  it needs to be placed as high up as possible.  What about in or around the lift room.  Yes it will be a bit noisy but think off that as the positive feedback that its working.

One generator is seldom enough to ensure absolute uninterrupted power.  Because of its mechanical nature and unregular use it will be prone to not starting automatically when you need it the most.  Think of it as your nice weather sports/convertible car that always have a flat battery the day the sun shines.   A good and periodic generator testing regime will help, but If your office don’t have 24 hour onsite personnel that is trained in how to start it in an emergency, you will need a backup.  And with in an emergency I mean the maybe 10 minutes you have before your ups runs out of batteries. 

There is ways of connecting power to make 2 generators cross-feed the same grid, together or one or the other.  It’s complicated and your average electrician is not to eager about it, but it can be done even a reasonable cost.   

Very few companies goes this far unless they have experienced first-hand what a unscheduled power outage can do to today’s vulnerable systems.  Think of however that with more abnormal weather patterns any place could experience a flash flooding, and have you investigated every part of the way electricity take from its place of production to your building.  You wouldn’t want any of those transformers to go off with a bang leaving you in darkness for what could be days.  

What is a servers economical life

In some industries they see it as important that a computer hardware has a long lifespan.  I hear that in the oil industry they are looking to get 7 years out of their servers.  Is this realistic.  In my opinion no.  Anybody that expects the highest possible uptime should not expect more than 3years, at most 4years,  lifespan from a server, or other computer hardware with moving parts. 

There is a reason for that with most manufacturers the 3 year warranty is cheap, year 4 reasonable and the cost of year 5 can be accepted.  Any more than that and there is definitely no economy in it.  It’s cheaper to buy new.  The trouble is in most cases no1 the harddisks and no2 the power supplies.  No amount of guarantees can make them live longer.  All the guarantees do is ensure there is something there to swap them with when they do fail, and failure usually mean some sort of downtime. 

A problem is that no hardware manufacturer will give you a “guarantee to fix”.  Service contracts involve a guarantee to maximum time before engineer is on site, and often a second maximum time until part is on site.  In the old days these where one and the same since the service engineer had a ll the most common parts in the back of his car.  These days stock is smaller and the part might be a flight away.  Not good if your problems is during a volcanic ash crisis, or a 9/11.

I can see the attraction of 7 years if you are in the sample financial industry since there is requirements to keep financial data for 7 years.  It is however better to prepare a migration path early, because it will be needed. 

Another requirement when trying to extend the life of hardware is frequent monitoring.  And here I must insist on the necessity for physical presence in the computer room.  Those blinking lights will for the trained eye give early warning.  They are also much easier to see than a line of log hidden among thousands.  Today we see the opposite trend with many being proud of their dark data centres.  Dark here meaning no light due to no manpower on site.

The necessity to be prepared will increase as more and more backups are online or on secondary storage.  They will expire just as fast as other hardware.  In the old days tapes could last long if they where kept well and tested for failure at regular intervals.  Always keep 2 at least copies because tapes did have a high failure rate due to stretch.  To ensure recovery I always brought 3 sets to the dr centre.  One problem is to find something to restore them on when they are needed after several years. 

A diskdrive is a diskdrive and they will fail eventually due to that they have moving parts inside.   A possible solution can be the ssd that could be written to once and then , which would help against its so and so many writes sudden death syndrome.  Currently however due to its cost its not taken up in any large numbers. 

Ssd’s  is not a solution for the parts of servers/applications/databases that requires frequent rewrites.  It could even shorten the lifespan of the hardware requiring a higher rate of turnover.  There have in use for high turnover database logs been reports of whole shelves of ssd’s failing at once, after less than 2 years of service.

Forgotten about act of day to day IT operations

An important but easily forgotten part of a company is its day to day IT operations.  What create kudos is new stuff, like development.  This is where Executive management has its IT eye in a company whose core business is not IT.  This is a natural progress from that share value is seen to have all current  status built into its current pricing and only new announcements can afflict it in a positive way.

This often leads to the attitude that IT operations is just “your job” and achievements is treated thereafter.   Sometimes for a CIO the only way of getting cudos for the operations part is through saving money by outsourcing it.  No wonder 90% of their attention is on development.   

Problem is that today IT operations is at the core of every business.  Few can handle even the simplest task without their backend computers, networks and core infrastructure running perfectly.  When you outsource it you lose part of the control on a vital part of your business success.  It’s not until it goes wrong, and sometimes very wrong ref Ulster bank, that top management takes an interest.  This often leads to a negative aspect.  Every time IT gets attention it’s because something went wrong.  And since rewards are only given for positive news, day to day running of IT becomes a neglected non rewarding area.  And the most neglected part of all is often Disaster Recovery.  A part of IT that will always have a certain stigma over it, due to it’s need of negative thought.  An important key element non the less since its what prepares you for what every IT manager now is  inevitable;  a major system let’s call it hiccup.

    

It is sad that many non IT executives in businesses whose core activity is not IT related, almost takes pride of their lack of knowledge of IT.  They couldn’t express it so openly if it was about any other side of their business they lacked core knowledge.  

This leads to that the CIO/IT Directors role becomes a buffer between IT and the top executives.  And since it’s the top executives that hires said CIO they often end up with a “talker” rather than a doer  with real knowledge.  

Building downwards from there, the “talker” CIO will see too that he don’t hire anyone in lower ranks that can upstage him/her.  Alternatively will feel a need to ensure that anyone able to do such a thing has no channel of communication with the executive level.  Often an easy solution since the executive level has no interest anyway.

This is the type of scenario where massive problems eventually will bubble to the surface.  It also makes for an unhappy company with resentment growing through the lower ranks.

To avoid more pitfalls in the future the executive level  needs to get a better understanding for what makes their business tick on a day to day level.  Unless you have your core right, future success can not be assured, and even the new stuff will only give temporary success at best.   

The losers in the long run will be the shareholders. Long term the company that do not have control of their day to day operations, and IT operations is a vital part of this, will lose out, or fail spectacularly.