Tuesday, December 7, 2010

Integrating Websense with ISA 2006

I recently had the dubious “pleasure” of having to install and integrate Websense with an ageing ISA 2006 Server.  After lots of trials and tribulations (then it works, then it doesn’t, or it appears to work but doesn’t filter etc. etc.)  I finally had to throw in the towel and call Websense support.  Here is what I had to do to get it working:

  1. Install Websense (with Filtering module) on the ISA server, making sure to select "the “ISA Integration” option during installation
  2. Installed the Websense ISAPI filter as per this Websense guide
  3. Stop and disabled the Filtering Service
  4. Edited the wsMSP.ini file (located in %windir%\system32) on the ISA 2006 server and change the EIMServerIP value to match the IP of the Websense server (in this case my ISA server)
  5. Created a “ignore.txt” file as per the Configuring ISAPI Filter section in the Websense Installation Guide for Microsoft ISA Server
  6. Websense support then had me disable ALL authentication methods in ISA Server.  I did this via Configuration – Networks – Internal – Properties – Web Proxy – Authentication
  7. They then had me set up the following Access Rule in ISA Server: "All outbound traffic" TO AND FROM "Local Host" and "All Protected Networks"

Only after all this did it work as expected.  I have not had the opportunity to test this with ISA’s successor, Microsoft’s Threat Management Gateway, but I would imagine that the procedure should remain fairly similar.

Sunday, December 5, 2010

Backing up Hyper-V VMs using DPM with software VSS Providers

DPM is a terrific backup solution for the all-round Microsoft shop.  In my humble opinion it’s also one of the best ways to back up Hyper-V Virtual Machines.  It integrates beautifully with your SAN vendors hardware VSS providers, but if you don’t have the budget to splash out on the requisite SAN licenses, there is a way to do it in software.

The Challenge

When backing up VM’s residing on a CSV, you might run into the following DPM error message:

Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Unknown error (0x8007173d) (0x8007173D))

The Solution

  1. Run the DSConfig.ps1 script (included below) on any node in your CSV containing cluster
  2. This will create a file called DataSourceGroups.xml in the same folder that you executed the DSConfig.ps1 script
  3. Copy DataSourceGroups.xml to the %programfiles%\Microsoft DPM\DPM\Config folder on your DPM Server
  4. This file needs to be updated every time you create and protect a VM using DPM
  5. If a protection group has already been created for the virtual machines, execute the Modify Protection Group Wizard. If a protection group has not been created, create a new protection group and the job serialization described above will take effect.

DSConfig.ps1 Script

The updated script can be downloaded from here

This will allow you to get rid of the pesky alerts and streamline your backup process.  The optimum solution would be to use a hardware VSS provider, but this is not always financially feasible, or if you have an iSCSI SAN.

Tuesday, September 28, 2010

Securing HP Procurve Switches (or how to placate Auditors)

I was recently approached by an organization who suffered through an IT Systems and Security Audit.  Needless to say the audited highlighted a couple of holes in their system, and I was tasked with implementing the audit recommendations.  Part of my scope of work included securing their HP Procurve switches, this post will outline a couple of best-practices when securing your network equipment.  It is by no means exhaustive, but it should provide you with a good baseline.  All instructions assume you're logged into the switch and at the (config)# prompt

Defaults Configurations and Secure Alternatives

Telnet vs SSH
Telnet is an antiquated and insecure protocol; it sends all traffic across the wire in clear text. This includes user
names and passwords. Any attacker may eavesdrop on a Telnet session and obtain credentials.  Below are the commands detailing how to disable Telnet and and implement SSH.
  1. crypto key generate ssh
  2. ip ssh
  3. no telnet-server
HTTP vs HTTPS
ProCurve switches and routers can, by default, be configured through the web interface. The HTTP
interface has the same limitations as Telnet. It is therefore recommended that the HTTPS interface be configured and the HTTP interface be disabled, like so:
  1. crypto key generate cert 1024
  2. web-management ssl
  3. no web-management plaintext
TFTP vs SFTP & SCP
TFTP client and server should be disabled as they do not require any authentication. Secure File
Transfer Protocol (SFTP) and Secure Copy Protocol (SCP) are part of the SSH protocol suite. As such provide an encrypted session using public/private keys between client and server just like
SSH.  You configure it like so:
  1. crypto key generate ssh
  2. ip ssh filetransfer 
  3. no tftp server
  4. no tftp client
SNMPv1/2c vs. SNMPv3
SNMP version 2 is enabled by default.  SNMPv2 community names are sent across the network as clear text (similiar to Telnet).  SNMP version 3 overcomes these weaknesses via encryption.  Here's how to configure it:
  1. snmpv3 enable
  2. snmpv3 only
If, for whatever reason, you cannot run SNMP v3, it is possible to run SNMP v2 in restricted mode, i.e. provide Read-Only access to your switch:
  1. snmp-server community restricted 
It is also possible to completely disable SNMP altogether:
  1. no snmp-server enable
Authorised IP Managers
It’s possible to restrict IP access to the switch to certain management IP's only.  This is configured via the following command:
  1. ip authorized-manager <IP address> mask  
Whilst not the be-all and end-all of security, the above should go a long way to ensure a reasonably secure base line for your HP Procurve network equipment.

Tuesday, September 14, 2010

Exchange 2010 ActiveSync Issue (or the dreaded HTTP Error 500)

Exchange ActiveSync (EAS for us smart people) is a pretty rocking way to get Smart phones hooked up to your Exchange Server.  9 times out of ten it "just works" if you follow the docs, but sometimes it doesn't.  Otherwise I wouldn't writing about it now, would I?

I had a rather perplexing case where EAS only worked for some users.  On the non-working users the only error message displayed was the rather unhelpful HTTP Error 500.  The only suggestion the usually excellent www.testexchangeconnectivity.com could offer was an old article applicable to Exchange 2003.

Diagnosing

The next step was to scour the Application log on the Exchange Server, and lo and behold, the following error was logged:

Source: MSExchange ActiveSync
Event ID: 1053
Level: Error


Exchange ActiveSync doesn't have sufficient permissions to create the "CN=User Name,OU=Admins,OU=Employees,OU=People,DC=domain,DC=local" container under Active Directory user "Active Directory operation failed on server01.domain.local. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
".
Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations.


Righto.  So it would appear that to actually use EAS, information has to be written to the user account and to do this the Exchange Servers group needs to have permissions to the user object.  Further investigation revealed that the only accounts giving the error was Admin accounts.

AdminSDHolder and SDPROP are the culprits

Because the failing accounts has domain admins rights their User Object permissions were being reset by
AdminSDHolder, via a process called SDPROP.  I quote "Each Active Directory domain has an object called AdminSDHolder, which resides in the System container of the domain. The AdminSDHolder object has a unique Access Control List (ACL), which is used to control the permissions of security principals that are members of built-in privileged Active Directory groups (what I like to call "protected" groups). Every hour, a background process runs on the domain controller that holds the PDC Emulator operations master role. It compares the ACL on all security principals (users, groups and computer accounts) that belong to protected groups against the ACL on the AdminSDHolder object. If the size or the binary string is different, the security descriptor on the object is overwritten by the security descriptor from the AdminSDHolder object.."

Resolution

Because SDPROP runs every hour, the fix needs to be applied and a EAS sync done before an hour has gone past.  Here is how:
  1. Launch the EMC from your Exchange CAS Server
  2. Browse to Server Configuration and select your server
  3. Go to the System Settings tab to see which Domain Controller is in use
  4. Launch ADUC from the DC located in step 3 and make sure Advanced View is enabled
  5. Right-click the user account and go to Properties
  6. Click the Security tab, then click Advanced
  7. Check the Include inheritable permission.... checkbox
  8. OK out of all the dialog boxes
  9. Sync your device
The sync should be successful, providing not more than an hour has passed between the sync and step 1.  Whilst this has been a very difficult issue to nail down, there is a lesson in there somewhere...DON'T USE PRIVILEGED ACCOUNTS FOR DAY-TO-DAY USE!!!

Monday, August 30, 2010

ESX Hosts Disconnecting After Upgrade to vSphere 4.1

When you upgrade to vSphere 4.1 your hosts might start disconnecting from your vCenter Server with the following error message: A general system error occurred: internal error: vmodl.fault.HostCommunication.  Restarting the management agents does not resolve the error, nor does rebooting the host.  This VMware KB points to name resolution issues, but that is not at fault here.  The issue is vCenter Server cannot manage an ESX 4.1 host.

Workaround / Solution

Currently there are two solutions available:
  1. Upgrade your vCenter Server to version 4.1.  (Once you've upgraded you'll have to remove the hosts from your inventory and re-add it - simply reconnecting didn't work in my case)
  2. Downgrade your ESX hosts to version 4
Strangely enough I could not find this documented anywhere on the VMware Knowledge Base, even though it seems to be a pretty widely reported problem.

Upgrading to vSphere 4.1 via a SSH CLI Session

I was tasked with upgrading a clients vSphere installation from vSphere 4.0 to 4.1.  Due to various external factors the client couldn't make use of the vSphere Update Manager, so I had to do it old-school style from the command line.  Here's how to do it:


Download the required updates
  1.  Navigate to http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4
  2. Download pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
  3. Download  upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
Install the updates
  1. Put your ESX host in maintenance mode with the following command: vimsh -n -e /hostsvc/maintenance_mode_enter
  2. Install the pre-upgrade patch:  esxupdate update --bundle=pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
  3. Install the actual upgrade patch: esxupdate update --bundle=upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
  4. Reboot the ESX host via the reboot command
  5. Last step is to exit maintenance mode: vimsh -n -e /hostsvc/maintenance_mode_exit
 All of the above can of course be automated using the Update Manager, but for those occasions where it's not possible to use it the above will come in handy.

Friday, August 20, 2010

Truncating SQL 2005 Log Files

Had a fun scenario recently where a customer (a Casino!) ran out of disk space on their SQL 2005 Server.  This in turn caused all their slot / video poker / whatever gambling machines to stop working.  Turns out old Bill Shakespeare had it all wrong, because it turns out that Hell Hath No Fury Like A Gambler Scorned!

Long story short - their SQL log file grew to gargantuan proportions, and yours truly had to whip them back into shape.  Here's how it went down!

  1. Run the following stored procedure: "use your_db_name"
  2. Followed by: "exec sp_helpfile".  This will return the physical names and attributes of files associated with your DB.  Record the DB and log filenames, without the path and extension
  3.  Enter the folowwing commands
    1. USE your_db_name
    2. GO
    3. BACKUP LOG your_db_name WITH TRUNCATE_ONLY
    4. GO
    5. DBCC SHRINKFILE (your_dblog_filename, 1) 
    6. GO
    7. DBCC SHRINKFILE (your_db_filename, 1)
    8. GO
    9. exec sp_helpfile
Step 9 should output the same info has Step 2, you can now compare the filesizes to see if the process was succesfull.

You might get an error "Cannot shrink log file because all logical log files are in use".  In that case you can follow the instruction here to resolve.  I've detailed the steps below, if you're too lazy to follow the link.
  1. Open SQL Enterprise Manager
  2. Right-click on the database you want to shrink and click Properties
  3. from the Data Properties go to Options.
  4. Set the Recovery Model to Simple and click OK and try to shrink the database
Your Database and Database log files should now shrink succesfully!

Thursday, August 19, 2010

Using Veeam backup to relocate VM's

Nope, I didn't fudge up the title.  Veeam Backup and Replication is a wonderful product, allowing you to replicate vSphere VMs to a offsite Disaster Recovery location.  When disaster strikes, it's a pretty straight-forward process to fail over to your DR site.  It's what I call a forehead procedure - you only have to hit the spacebar with your forehead.    Thanks - I'll be here all night!  

What's not so intuitive and well documented is using Veeam Backup to move VM's to a different location, for example a Server Room / Data Center relocation *and then commiting those changes*, i.e. not failing back to Production.  The below steps assume we've replicated and failed over our VM's to our DR location already.

Ruan's Step By Step Guide on using Veeam Backup to relocate VM's

  1. Delete all Veeam VM snapshots using the vSphere Snapshot Manager
  2. By default your DR replica will be named "VMname_replica", rename it back to its original name, i.e. VMname
  3. Remove the VM replica from the list of replicas in the Veeam Backup and Replication Console
  4. Delete the Production -> DR replication job responsible for replication the VM in question.  Recreate it to reflect the new Source and Target locations
  5. Delete the .vrb file from the VM datastore, as we will no longer be using these restore points
  6. Delete the replica.vrb and running.rbk files
  7. Pat yourself on the back - you've just done the easiest VM move you'll ever do!

Thursday, July 29, 2010

The Overheating EVA

I had a "interesting" experience recently, namely had a EVA 4400 overheating due to environmental issues (fancy-talk for Aircon failure).  The client phoned me, complaining that half of their Hyper-V VM's are not running.  Further investigation revealed that the CSV's were offline.  Hmmm, this was getting serious.  I logged into Command View and saw that most of my VDisks were faulted, this was due in no small matter to the fact that all the drives in one of my shelves were faulted.

Event Logs

I had a look at the relevant EVA logs and discovered the following relevant entries:
  •  Temperature within an HSV300 controller becoming too hot.
    View corrective actions.  Corrective action code: 2e
  •  A drive enclosure temperature sensor out of range condition has been reported by one of the drive enclosure link modules.
  • A physical disk drive has disappeared.
    View corrective actions.  Corrective action code: 42
  • A Volume has transitioned to the MISSING state.
    View corrective actions.  Corrective action code: bf
What Happened

In retrospect it was a fairly simple sequence of events, as evidenced by the entries above.  The Air Conditioner failed, which caused the temperature within the Drive Shelf to rise (this is the HSV300 controller referred to in the event log).  To prevent damage to itself, the drive then switched itself off, which prompted the log entry about the physical drive disappearing.

We then started seeing volumes transitioning to the missing state, i.e. our VDisks went missing.  Hardly surprising considering that the drives containing them switched themselves off.

Resolution

  1. Restored Air Conditioning (goes without saying I guess)
  2. Powered off the EVA and all attached disk shelves
  3. Powered on disk shelves and waited for the Numeric ID LED's at the back to display the proper IDs.
  4. Powered up the Controller
  5. Lo and behold!  All the previously failed physical disks came on-line, meaning that my missing VDisks also made a most welcome return
  6. Unfortunately my Hyper-V Hosts still couldn't access the Vdisks, so I had to unpresent and re-present them via Command View.  I assume the EVA assigned new WWN's to the LUNs.
  7. I re-scanned for storage from the Disk Management MMC on the Hyper-V Hosts
  8. Brought the Disks and CSV's online via cluster manager
  9. Started up the VM's
Conclusion

This was quite a harrowing experience, obviously.  What struck me as ridiculous is that HP does not have *ANY* thermal shutdown logic / capabilities on the EVA controller itself.  It keeps on trucking till the drives themselves fail, causing a very ungraceful failure of the VDisks.  There is also no guarantee that your drives and VDisks will come back online.  In essence - if your EVA overheats there is a distinct possibility that you lose your Data.  Caveat Emptor...








Friday, July 2, 2010

Move WSUS server to a new server

Namibia is a third world....errr....developing country.  So apart from rampant (55%) unemployment it also means that we are bandwidth starved.  This also means that some businesses are running on capped accounts, so attempts need to be made to conserve bandwidth.  I've had to move WSUS servers to new hardware or VM's a couple of times, and needless to say it's a huge time-sink and waste of bandwidth to re-download all updates every time you move your WSUS server.  Thus I've come up with a way to move WSUS without downloading tons of patches - steps are outlined below for your enjoyment.

  1. Install WSUS on your new server, making sure to select the option to use the existing Windows Internal Database
  2. During the Choose Upstream Server of the configuration wizard, be sure to select Synchronise from another Windows Server Update Services Server
  3. Ensure that the This is a replica of the upstream server check box is selected.  This ensures that existing approvals, settings, computers and groups are maintained
  4. Complete this Wizard
  5. Your new replica server will synchronise with your upstream server.  This is what we're talking about - no re-downloading many GB's worth of patches!  N.B.  Wait for this process to complete before carrying on with step 6.
  6. Now change your Update Source and Proxy Server settings to Synchronise from Microsoft Update
  7. Now for the magic bit.  Download and install the WSUS 3 API Samples and Tools on both your old and new WSUS servers
  8. Open up a CMD Prompt and navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationExport" folder on your old WSUS server
  9. Run "wsusmigrationexport.exe WSUS_Settings.xml" to export the settings. This will backup your approvals and target groups to an XML file
  10. Transfer the WSUS_Settings.xml created above to your new WSUS server
  11. Again navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationImport" folder (on the new WSUS  server). Run "wsusmigrationimport.exe WSUS_Settings.xml All None"
  12. Review and compare settings on your two WSUS servers, ensuring that they match each other
  13. Update the relevant GPO's to ensure clients are pointing to the new WSUS server
One last point, I've noticed that if you are not using GPO's to assign your computers to Computer Groups all your clients will get stuck into the Unassigned Computers group.  You'll have to manually sort them into the appropriate groups again.  Bummer, but you really should be using Computer Groups....

Monday, June 21, 2010

Back to Basics - Create a LUN and present it as a Cluster Shared Volume

You know when you do something and you think to yourself "Oi, I should really document this"?  And then you go "Nahhh, this is easy peasy lemon squeezy - I won't forget" and a month later you have to repeat the task and wonder of wonders, you did forget.  Well Goddammit - never will I have to wonder again how to create a LUN on a HP EVA 4400 SAN and use said LUN (or Vdisk in vHP's vVirtual vSpeak vWhere vEverything vStarts vWith a vStupid vV). vWhere...er I mean where was I.  Ahh yes, lets create a vDisk...

Create a vDisk on the EVA SAN
  1. Log into Command View (what a cool name, sounds like you can bomb some backwater country into the Stone Age with a mere misplaced mouse-click)
  2. Expand your Virtual Disks Folder
  3. Click Create vDisk
  4. Name your disk, then give it a size and RAID level
  5. Click Create vDisk
Present the vDisk to your Hosts
  1. Navigate to your newly created vDisk and click the Presentation tab
  2. Click Present and select all hosts to present the vDisk to.  For some obscure reason every 4400 I've worked on would only allow me to present to four hosts / HBA's at a time, anything more and it errors out
Initialise and format the vDisk
  1. Open the Disk Management MMC on one of your SAN-connected Hyper-V hosts
  2. Right-click the Disk Management node and click Rescan on the context menu
  3. Right-click on the newly appeared unallocated disk and select Bring Online
  4. Right-click the disk and click Initialize
  5. Right-click (how's that middle finger doing?) on the unallocated storage and click Create Simple Volume
  6. Complete the wizard, but do not assign a drive letter
Create a Cluster Shared Volume using our vDisk
  1. Open Failover Cluster Manager
  2. Go to the Storage node and click Add a Disk in the Actions pane
  3. Select the disk you wish to add and click OK
  4. Right-click the Disk and click Properties - now give it a descriptive nameBecause I say so
  5. Go to Cluster Shared Volumes and click Add Storage in the Actions pane
  6. Select the disks you wish to add and click OK
  7. The CSV will now magically appear in your ClusterStorage namespace.  Navigate there and give it a descriptive name
  8. You've just greated a CSV full of win, which can now host a highly-available VM! 

Friday, June 4, 2010

The low-down on VLAN's

I find myself having to explain the concepts behind VLAN's a lot, so I've decided to do a blog post about it.

The Wikipedia article defines a VLAN as: "a group of hosts with a common set of requirements that communicate as if they were attached to the same broadcast domain, regardless of their physical location. A VLAN has the same attributes as a physical LAN, but it allows for end stations to be grouped together even if they are not located on the same network switch. Network reconfiguration can be done through software instead of physically relocating devices".

PROTOCOL CONCEPTS

We have regular "untagged" Ethernet frames. ("Frame" is the data-link (layer 2) term for what some protocols calls a "packet" or "datagram".) These are the same Ethernet frames from before VLANs were invented.

Then we have "tagged" frames. These take a regular Ethernet frame and add a marker -- the "tag". The tag itself contains the VLAN number (along with a priority code). Thus, a tagged frame is associated with a particular VLAN number.

Inside the frame, the tag is basically just a reserved Ethernet frame type, followed by the tag information, followed by the rest of the original frame (complete with its original frame type). So to network nodes which are *not* VLAN aware, a tagged frame looks just like a regular Ethernet frame, but with an "unknown" frame type.

For computers, a VLAN-aware network stack will usually turn each VLAN into a separate network interface, as if each VLAN was a different network card with a different cable. The "stock" network stack included with Win 2000/XP does *not* include VLAN capability, so they can only handle untagged frames. Tagged frames will generally be silently ignored. Third-party software (often included with high-end network cards) can add VLAN capability. I'm not sure about 2003/Vista/2008.

Switches which are not VLAN-aware, along with any repeaters, will treat tagged frames exactly like they do any other frame.

SWITCH CONCEPTS

Switches which are VLAN-aware generally let you control which ports are associated with which VLAN(s). The terminology, commands, etc., will vary with manufacturer. I'm most familiar with the HP ProCurve managed witches, so I'll speak mostly to that. The concepts should carry over to other manufacturers.

With HP, you can tell the switch that each VLAN is tagged, untagged, or not available for each port. Internally, frames are *always* associated a VLAN number. The configuration just tells the switch which VLANs go with which ports, and which VLAN number to use for untagged frames.

When the switch receives an untagged frame, it needs to know what VLAN to associate it with. If the port has an untagged VLAN associated it, the fame will go to that VLAN. If there is no untagged VLAN associated with the port, the frame is dropped. This is why a port can have (at most) one untagged VLAN associated with it -- how could the switch know which one to use otherwise?

When the switch receives a tagged frame, it simply checks to see if that VLAN is associated with the port. If so, it accepts it; if not, it drops it. This is basically just an access control mechanism

When the switch has a frame to *transmit*, it only considers ports which are associated with the frame's VLAN. If a port is configured as tagged for the frame's VLAN, it transmits the frame with the VLAN tag. If a port is configured as untagged for the VLAN, the frame is transmitted on that port without a VLAN tag.

Real-World Implementations

Okay, enough conceptual stuff, how do we use it? Below I'll describe a implementation I had to do for a client recently.

VLAN's were numbered and split up as follows -

1 – Reserved

2 – Main company private network

3 – Guest network

4 – ISP uplink

5 – IP telephony

I used separate VLANs for security, manageability, and robustness. I don't want guests to be able to probe/sniff/screw-up the corporate network. I didn't want the cable modem link (which is full of garbage traffic from other subscribers) on the main LAN. I want the phones to have higher priority over regular data, and to have their own DHCP server, so if the "data side" of the house goes down, they still have phones.

I don't, as a rule, use VLAN 1 because that's the default on most things. This also avoids confusion.

VLAN 2 is where most traffic is in this scenario. Most ports are configured as untagged for VLAN 2. The computers, printers, etc., plugged into those ports aren't even aware they are on a VLAN. All the employee workstations are like this, as are most servers. The syntax to configure this is easy. For example, if I want port 1 to be untagged for VLAN 2 like this:

vlan 2 untagged 1

VLAN 3 is for guests (visitors). For obvious reasons, I don't let them plug in to the main network. They get a separate IP subnet, a separate DHCP server, and bypass much of the network filtering. Again, all untagged, and connected nodes are not aware of the VLAN.

VLAN 4 was created because the client has a ADSL Internet feed they use for employee web browsing. Circumstances meant the modem had to go in a different building from the main server/network room. So the cable modem plugs into a particular switch port, which uses VLAN 4 to carry it to the firewall in a different building. Again, untagged, unaware.

VLAN 5 is for premises voice-over-IP. The phones include a built-in switch for "daisy chain", and are VLAN-aware. Switch ports connected to phones are tagged for VLAN 5 and untagged for VLAN 2, so you can plug a PC (not VLAN aware) into the phone. Traffic is kept separate, and voice traffic is given priority. The central phone equipment has an Ethernet port which we connect to one of the core switches, with that switch port untagged for VLAN 5. So the central equipment just thinks it is talking to a regular LAN.

For example, suppose port 20 was the central phone equipment, and port 7 had a VoIP phone. We would do:

vlan 2 untagged 7

vlan 5 tagged 7

vlan 5 untagged 20

For links between switches, each VLAN is configured as *tagged* for those ports, with *no* untagged VLAN. Those ports are only used to carry traffic between VLAN-aware switches. (Cisco would call this a "trunk port", IIRC.) The HP syntax for (e.g.) port 24 as a switch link would be:

vlan 2 tagged 24

vlan 3 tagged 24

vlan 4 tagged 24

vlan 5 tagged 24

Finally, our firewall, which *is* VLAN-aware, is configured as tagged for all VLANs on its port, with no untagged VLAN. Virtual network interfaces are configured in the firewall OS, and all the IP configuration and filtering rules are done with those virtual interfaces, not the physical interface. This also means our firewall only has one network cable plugged into it. Configures the same as a switch link, really.

Hope this helps someone!

Monday, May 31, 2010

How to add a Node to a Windows 2008 R2 Hyper-V Failover Cluster

Dammit - me and my lack of documentation. Worked with Hyper-V for the first time six months ago (I'm predominantly a VMware dude). I Had to add a node to a failover cluster this morning, and found myself wishing I had documented it better the first time around. Got it sorted after a bit of scratching around; as you'll see below I've also finally gotten round to documenting the process. The below assumes that you've hooked it presented appropriate the appropriate LUNs on the SAN etc. etc....

  1. Install the OS (no sh!t Sherlock). Make sure you run the same edition on all nodes. Also installed any required MPIO software that may be required by your SAN.
  2. Join the server to the same domain that your other Cluster nodes are members of
  3. Enable and configure Windows Updates
  4. Make sure that your network adapter naming convention is the same as on the other nodes. This will make management easier
  5. Install the Hyper-V role, reboot when prompted
  6. Make sure to have naming consistency in your Virtual network. In other words, exactly the same name and capitalization
  7. Install the Failover Clustering feature, reboot when prompted
  8. Physically connect the server to your SAN
  9. Join the node to the cluster. This can be done via the Failover Cluster Management MMC: Right click your cluster name, select Add Node, enter the server name and click OK. The command line alternative is "cluster.exe /cluster:Cluster_Name /add /node:Node_Name"
It might be necessary to change your quorum settings afterwards - this Microsoft webpage explains the options rather nicely. The quorum settings are changed via the Fialover Cluster Manager MMC, in the Actions pane there's a Configure Cluster Quorum Settings link.

How to Expand a VHD on Hyper-V 2008 R2

This is a fairly basic task that needs to be done fairly frequently, but I've never documented it anywhere so it always takes me 10 minutes longer than what it should. The client where I had to do this today happens to run a HP EVA 4400 Fiber SAN, so I will include the instructions for the EVA side of things as well. Without further ado...

Expanding a existing LUN on an HP EVA 4400 SAN
  1. Log into Command View (HP's delightfully named SAN configuration solution)
  2. Navigate to the the Virtual Disks folder, more specifically to the Vdisk you want to expand
  3. Click the Vdisk, and wait 10 seconds for the incredibly laggy Command View interface to catch up
  4. Type your new, preferred size in the box cunningly labelled "Capacity"
  5. Click "Save"
Expanding a VHD
  1. Use Cluster Resource Manager to determine who owns the Disk Resource you wish to expand
  2. Once identified, go to "Disk Management" on the owner
  3. Right-click the Disk Management node, and select "Rescan Disks". The un-allocated space should show up now
  4. Right-click the volume you wish to expand and select "Extend Volume". Click Next 2 times and finally Finish
  5. Fire up SCVMM and shut down the VM whose VHD you want to expand
  6. Click the "Hardware Configuration" tab, and select the Disk you want to expand
  7. Check the "Expand this disk" checkbox and enter the desired size
  8. Click OK
This should take care expanding the VHD, but this does not mean the underlying OS will automatically make use of the added capacity, no sirrreeee. In my case the VM was running Windows 2008 Server R2, and this is what I had to do....

Using Diskpart to Extend a Partition
  1. Click Start, type CMD and press Enter
  2. In the resulting Command Prompt, type "Diskpart"
  3. Select the correct drive and partition to perform the extend operation on. In my case it was Disk 1, Partition 1. The correct commands in my case were "Select disk 1" and "Select Partition 1". The "List disk" and "List partition" commands, surprisingly enough, will list your existing disks and partitions
  4. Now that we've got the correct disks and partitions selected, we simply give the "Extend" command. This will extend the disk using all the contiguous free space available
And with that done, you should be the proud Admin of a newly extended VHD!

Monday, March 29, 2010

Enabling Jumbo Frames in vSphere

Hi Kids, todays post is brought to you by the letter J. I previously mentioned a vSphere deployment I had to do, connecting to a EMC AX4-5i iSCSI SAN. Once I got the storage hooked up to my ESX hosts I of course wanted to enable jumbo frames.

Jumbo Frames, for the uninitiated heathens out there, is basically a Ethernet frame with a payload of more than 1500 bytes, up to a typical maximum of 9000 bytes per frame. Why would I want to do this? Performance, in a nutshell. The only requirement would be that your switch supports Jumbo Frames, which I believe most, if not all, mid to high-end kit does. So let's get down to business, shall we?

Switch Configuration

I'm fairly proficient with Cisco and HP networking gear, so I'll give a quick rundown of the commands needed to enable jumbo frames on their kit:
  • Cisco: Go into conf t mode and enter the following command "system mtu jumbo 9000". Once that is done issue a "reload" command to reboot your switch
  • HP Procurve: On these babies Jumbo frames need to be enabled per VLAN. Execute the following command: "vlan # jumbo"
That takes care of the networking side of things, let's move on to the ESX...

Configure Jumbo frames in vSphere
  1. Log on to your ESX host using your favorite SSH client
  2. Change your chosen vSwitch MTU with the following command "esxcfg-vswitch -m 9000 vSwitch#" Replace vSwitch# with the name of the vSwitch you want to modify
  3. Seeing as one cannot change an existing VMKernel port MTU, you will either have to remove and recreate your existing VMKernel port, or create a new one. To delete an existing VMKernel port, enter the following command: "esxcfg-vmknic -d -p VMKernelport"
  4. Now let's add a VMKernel port called "iSCSI01" to our vSwitch, like so: "esxcfg-vswitch –A iSCSI01 vSwitch#"
  5. Now we enable jumbo frames on our VMKernel port: "esxcfg-vmknic –a –i 192.168.x.x –n 255.255.255.0 –m 9000 iSCSI01". You do know you need to change the IP address and mask to reflect your environment, right?
  6. Let's make sure we didn't screw up somewhere by running "esxcfg-vmknic –l". Verify that the MTU is set to 9000
  7. All done! You can refer to my previous iSCSI Multipathing post to ensure Jumbo frame enabled iSCSI Multipathing goodness!
I have thought of merging this post with the iSCSI Multipathing post, but seeing as not all environments can/will support end to end Jumbo frames, I have decided to keep them seperate for the time being.


Friday, March 26, 2010

RPC Endpoint 6001, and why I couldn't ping it

Look, I love Exchange 2010. As a matter of fact I'm pretty sure Bono is busy installing it somewhere in a attempt to cure World Hunger. It's like caviar, dipped in celery salt and served from Uma Thurman's belly button...but geez Louise, it can be a frustrating beast at time as well, sometimes through no fault of it's own, as we'll see below.

There my Exchange 2010 was, humming along nicely, I already moved the mailboxes across, configured Autodiscover, Outlook Anywhere, ActiveSync, all those highly visible, nice to haves. I created a test account, tested all the remote services, and it worked beeeautifully! So with a flourish I sent a mail to the client, giving him the go-ahead to roll out the remote / mobile services to his users. Only to hear back from them 10 minutes later that bugger-all remote services works. Godammit. No OA, no AS, no OWA, nada, zip. After wiping the egg from my face, I started troubleshooting. First instinct was that I somehow fudged my ISA Publishing rules, but no, it was all good. I then headed off to the bestest Exchange Connectivity Troubleshooter in the history of the world. I ran a test or two and it told me in no uncertain terms that:
Attempting to ping RPC Endpoint 6001 (Exchange Information Store) on server exc.clientabc.local
Failed to ping Endpoint
Tell me more about this issue and how to resolve it
Additional Details
RPC_E_ACCESS_DENIED error (0x5) was thrown by the RPC Runtime

Now the interesting thing here was the fact that it was only doing this for existing users, newly created ones worked without a hitch. I had a antacid (for some reason troubleshooting permission issues on IIS gets to me) and got to work. Suffice to say that I compared IIS permissions, folder NTFS permissions, even user properties in ADSI Edit. I ensured the new users had exactly the same group membership as the existing ones, everything. Then my eye caught something in the security log, All existing users, when logging on via any remote services, threw a EventID 534 in the Security Log; The user has not been granted the requested logon type on this computer.

I was making progress! a Quick review of the existing users account properties revealed that users were only allowed to log onto their workstations. The new test accounts that I created had no such restriction of course. Once I removed these restrictions from the user accounts everything worked flawlessly!

P.S. I tried granulizing (is that even a word?) the logon restrictions by allowing the users to logon to the Exchange 2010 Server. This fixed the OWA, but OA and AS was still not working. The only way I could get OA / AS to work was to remove all logon restrictions.

Missing Intermediate Certificate in Certificate Chain

As promised, I will outline some of the issues I had with a recent Exchange 2010 migration. After Installing the required UCC and intermediary certificates on both the Exchange and ISA 2006 SP1 servers, following the GoDaddy instructions here, I ran into some issues. Namely ActiveSync refused to work. I headed of the best Exchange Connectivity Troubleshooting site EVAR! This gave me the following detailed info:

Validating certificate trust for Windows Mobile Devices
Certificate trust validation failed
Additional Details
Missing intermediate certificate in Certificate Chain. Subject =
SERIALNUMBER=07969287, CN=Go Daddy Secure Certification Authority,
OU=http://certificates.godaddy.com/repository, O="GoDaddy.com, Inc.",
L=Scottsdale, S=Arizona, C=US, See KB 927465 for more details.

After some serious troubleshooting, googling etc, I still didn't have the issue fixed. As a last resort I rebooted (thought those days were gone) the ISA server. Lo and behold it worked, but that's four hours of my life I'm not getting back anytime soon. What's weird about this is that it's the first time I've had to reboot a ISA server to get a certificate to "take". Oh well...

Thursday, March 25, 2010

ESX 4 iSCSI Multipathing

There I was, setting up a small DR site for a client, hardware was a blinged up Proliant DL380, connecting to a sweet, sweet little EMC AX4-5i SAN. This being my first non-FC SAN, I obviously investigated the multipathing options. I was kind of (no, really!) taken aback when I discovered that it's not so straightforward to set up. What's a boy to do? I took a deep breath, bowed in the direction of the (not-so-sweet-anymore) AX4-5i SAN and unleashed the fury of my Google-Fu! Hee-Ya! About 3 hours later the fiendish ESX 4 and a AX4-5i had to yield to the fury of my touch typing - below I chronicle it for all eternity.

Setting up Multipathing
The DL380's I worked on here had an additional 4 port NIC installed, for a total of 8 1Gb ports. The 4 on-board ports I dedicated to the Service console, vmkernel et al, standard stuff. The 4 ports on the add-on NIC I used for iSCSI, perfect considering the AX4-5i also has 4 iSCSI ports. What I did then was set up a new vSwitch, added my 4 Nics to the vSwitch and then created 4 seperate VmKernel ports, so that we have a one-to-one mapping between the VMkernels and the NICs. Here's how to do that:

  1. Create a new vSwitch that we're going to dedicate to iSCSI
  2. Connect your NICs to this vSwitch by going vSwitch - Properties. Click the Network Adapters tab and click Add. Select your NIC's and click Next and Finish.
  3. Now we create VMkernel ports for all our newly added adapters, like so: Go vSwitch - Properties and click the Ports tab and click Add. Select VMkernel, give it a nice label and a suitable IP address and click Finish. Create a VMkernel port for every adapter assigned to your iSCSI vSwitch.
  4. Now the important bit, we're going to do the one-to-one VMkernel to NIC mapping: On the ports tab, select one of your new VMkernel port and click Edit.
  5. Select the Nic Teaming Tab and check the Override vSwitch failover order
  6. Make sure only one adapter is active, move all the other NIC's down to the Unused Adapters section
  7. Rinse and repeat for every VMkernel port on your vSwitch
Alas, we're not done yet - now I'm going to introduce you to my little friend esxcli. This fun little guy will enable us to connect our newly created VMkernel ports to the ESX iSCSI initiator

Enable Software iSCSI Multipathing
  1. From the CLI (console, ssh, pick your poison) run the following command esxcli swiscsi nic add -n vmk# -d vmhba33
  2. Rinse and repeat for all your ports, verify your work by executing the following command: esxcli swiscsi nic list -d vmhba33
  3. You can then rescan your iSCSI initiator via the vSphere GUI. You can also verify multpathing via the paths view for the vmhba33 adapter.
Happy Multipathing! It should be mentioned that iSCSI and Jumbo frames go hand-in-hand. I will do a post detailing that can of worms in the near future.

Exchange 2010 SAN / UCC Certificates Installation

So, I recently completed a Exchange 2003 to Exchange 2010 migration for a mid-sized client. There were a (of course) couple of hiccups, which I will detail as soon as I get my stuff in order. With that said, here's what I learned about UCC certificates (more specifically GoDaddy UCC Certs).

THE BASICS:

This client had 2 separate DNS namespaces, the AD DNS was clientabc.local, the external DNS was clientabc.com. Internally the Exchange server was called cabc-exc-001, externally it was mail.clientabc.com. So let's get down to the 3 commandments.

  1. Any name by which your server will be accessed needs to be included on the certificate. In my case it was the following: mail.clientabc.com, cabc-exc-001.clientabc.local and finally clientabc.com
  2. Make the common name the server's external DNS alias, eg. mail.clientabc.com
  3. If you use the Autodiscover server (which you should, it RAWKS) you should add that to your UCC certificate. In my case: autodiscover.clientabc.com and autodiscover.clientabc.local
Generating the Certificate Request:

  1. Fire up your EMC and click "Manage Databases" on the homepage
  2. Click "Server Configuration", then click on "New Exchange Certificate" in the actions pane
  3. You'll be prompted for a "Friendly Name". This is purely descriptive, so call it something descriptive.
  4. On the "Domain Scope" dialog, do not select the "wildcard" option
  5. Next up is the "Exchange Configuration" menu. Check the boxes for the services you plan to secure. The wizard will recommend names, ensure they're correct for your environment, keeping in mind our 3 commandments
  6. On the next screen you'll be allowed to enter your Org info
  7. Et viola! Click on the "Browse" button to save all hard work from above into a .req file
  8. The contents of the .req file must now be submitted to your Certificate vendor of choice (I used Godaddy).
  9. Once you've completed that you should be able to download your certificate. Once that is done it's on the next section.
  10. It's of course also possible to do all of the above via the EMS. Using my example the command would be: "New-ExchangeCertificate -GenerateRequest -KeySize 2048 -SubjectName "c=NA, s=Erongo, l=Swakopmund, o=ClientABC, ou=Information Technology, cn=mail.clientabc.com" -DomainName cabc-exc-001.clientabc.local, autodiscover.clientabc.com, autodiscover.clientabc.local, clientabc.com -PrivateKeyExportable $True"
Installing Your UCC Certificate
  1. Download and save the certificate from your provider
  2. Now install any intermediary certificate, following instructions provided by your chosen CA. THIS IS CRUCIAL! Install this before you install your actual certificate.
  3. Now start up the EMC again and click "Manage Databases" on the homepage. Click "Server Configuration", then select your certificate.
  4. In the Action Pane, click on "Complete Pending Request"
  5. Browse to your downloaded certificate, and click Open, Complete and Finish.
  6. From the Action Pane, click "Assign Services to Certificate", select your server from the list and click Next
  7. Select the necessary services, then click Next, Assign and Finish
  8. Alternatively we can import our certificate with a EMS command: Import-ExchangeCertificate -path c:\certreq\mail.clientabc.com.crt -friendlyname "Your Friendly Name"
  9. Then assign the services like so: Enable-exchangecertificate –services IIS –thumbprint
To make sure everything takes, you can restart the Exchange Transport Service. And that, as they say, is that!