Build an ESXi server on the Cheap

If your a network admin, and you’re not using virtualization in your disaster recovery, then you’re probably being negligent. VMware ESXi is free. You can’t do all of the fancy stuff, like VM motion, but it is the same bare-metal hypervisor as ESX. You can register, download, and obtain a free license for ESXi here.

ESXi is pickier when it comes to hardware than its pricier counterpart ESX. However, if your hardware works with the drivers it does have, you can add your hardware IDs to the configuration files and use your hardware. If you use a system, motherboard, and/or NIC listed on the community maintained list and download and use the community maintained configthen you can forgo a lot of tinkering. If you are going to use local storage, you won’t be able to use RAID with a SATA drive, or at least I haven’t figured it out; that would probably slow things down anyway. ESXi can use storage on NFS file shares, SANs, and some NAS devices as well. I am going to look at using FREENAS to hold some VMs instead of shelling out boo koo bucks for fiber. Whatever you decide to do, make sure you get an Intel NIC listed on the HCL; I had no luck with anything else.

For now, I am using a newly built device for DR testing and proof of concept. I went to my local Micro Center, friggin love that store, and purchased an ASUS PQ5 SE2 ($95), 4 GB 800mhz DDR2 Corsair RAM ($70), a 45nm 2.6 Ghz 8mb Core 2 Quad ($189),  and a 1TB WD SATA HD with a dual IO controller ($139). I happened to have a case/power supply, DVD drive, and compatible Intel NIC laying around. So I spent about $480 for the parts I needed.

I elected to install ESXi on a bootable flash drive.  VMware made a video  and pdf showing how. Here are the steps on how to do this on a Windows machine per yellowbrick.com:

  1. First get the following tools: 7-Zip(Free), WinImage(Demo)
  2. Download the ESXi ISO
  3. Open the ISO with 7-Zip
  4. Extract “install.tgz”
  5. Open “install.tgz” with 7-Zip
  6. Click on “install.tar”
  7. Browse to “usr\lib\vmware\installer\”
  8. Open “VMware-VMvisor-big-3.5.0_Update_2-103909.i386.dd.bz2″
  9. Extract “VMware-VMvisor-big-3.5.0_Update_2-103909.i386.dd”
  10. Open WinImage and go to Disk, click on “Restore Virtual Harddisk Image on physical drive”
  11. Select a physical drive
  12. Select “VMware-VMvisor-big-3.5.0_Update_2-103909.i386.dd”
  13. Click “yes” to write the DD image to the USB Disk

After building an ESXi bootable flash drive, I downloaded the community maintained config, renamed it to oem.tgz, and pasted into the bootable partition on the flash drive – overwriting the existing one.

Time to boot it. After inserting my USB ESXi, I had to go into the BIOS of my board and set the USB drive to be bootable. I also had to change my SATA drive configuration to be AHCI instead of RAID or IDE.  After saving, I was eventually greeted by the ESXi system.

You will need to adjust your network settings to be appropriate for your network, which you won’t have the option for unless you have a compatible NIC (get a listed Intel NIC). Once ESXi is network capable, you can go to your server’s IP address (http) and download VMware Infrastructure Client to manage it. I went to VMware’s site and downloaded VMware Converter (also free) to convert an existing workstation VM to ESX format; took a while but I did it over wireless.  You can also use the converter to convert an existing physical computer to VM. The converter uploads directly to your ESXi server, and it will complain about using a login without a password, so add a password to your ESXi root user account and use that.

esxi server

So far, its working like a champ. The next step is to set up a FREENAS iSCSI target server via crossover cable to a dedicated NIC.  If you have any pointers, let me know. Cheers.

Create an IDS on the Cheap

Having an Intrusion Detection System isn’t optional for network admins that are audited. An IDS is also very handy to have as it shows you everything that is going on. You don’t have to spend a lot of money on this, especially since most of the best tools are free. In my case, I wanted: network intrusion detection, network inventorying, and security scanning. I spent a lot of time messing around with separate packages for getting the functionality I wanted, but it was a pain to manage them all separately. I finally found an all-in-one system which met all of my requirements. OSSIM provides snort, nagios, ntop, and nessus (and various other security tools); this met all of my needs. It comes in the form of an installable debian linux image. Get it here.

I used an old PC as my server, adding an additional nic. When installing, I had to boot with –aspi=off since the power management settings on the board weren’t detected properly. After getting over that hurdle, I simply entered options applicable for my environment via the wizard that is provided. After the installation finished, I had to spend some time configuring. I made this guide piecing together info from various sources.

Update your linux patch level

Nothing fancy here. Its debian, so use apt to install patches

  1. ssh into the OSSIM server as the admin user
    #ssh root@192.168.1.3
  2. Update the package list, install patches, and reboot
    #apt-get update
    #apt-get upgrade
    #reboot 
        

Configure Network Interfaces

The LAN that needs monitored is 192.168.1.0/24. Simply change the LAN address to suite your network. The OSSIM server has two nics: one with an IP address (192.168.1.3) used for management and another operating in promiscuous mode (0.0.0.0) for sniffing. Configure a port on the main backbone switch, a gigabit smart switch which all of my servers and internet connection sit on, to be a spanning  or mirroring port of all other ports. This means that the port would receive a mirror of all traffic on all other ports; this is what the promiscuous nic is plugged into. The management interface (192.168.1.3) is plugged into a separate switch that is chained to my backbone switch. Change the OSSIM network configuration to reflect this setup.

  1. Edit the interfaces configuration using your favorite linux text editor; I like nano
    #nano /etc/network/interfaces
  2. Set the /etc/network/interfaces file to be the following, you would need to replace addresses with ones applicable for your network.
    auto lo eth0 eth1
    iface lo inet loopback
     
    # The primary network interface
    # Used for web management
    iface eth0 inet static
            address 192.168.1.3
            netmask 255.255.255.0
            network 192.168.1.0
            broadcast 192.168.1.255
            gateway 192.168.1.1
            # dns-* options
            dns-nameservers 192.168.1.8
          
     
    #eth1  is for monitoring the LAN A network
    # no ip address, promiscuous mode
    iface eth1 inet manual
            up ifconfig $IFACE 0.0.0.0 up
            up ifconfig $IFACE promisc
            down ifconfig $IFACE down
  3. Save and close the file. Restart the network interfaces
    #/etc/init.d/networking restart 
  4. Configure ntop to use the promiscuous nic
    #dpkg-reconfigure ntop 
  5. Configure snort to use the proper interface
    #nano /etc/snort/snort.debian.conf

    Set the /etc/snort/snort.debian.conf to be the following:

    DEBIAN_SNORT_HOME_NET="192.168.1.0/24"
    DEBIAN_SNORT_INTERFACE="eth1"
    DEBIAN_SNORT_OPTIONS=""
    DEBIAN_SNORT_SEND_STATS="true"
    DEBIAN_SNORT_STARTUP="boot"
    DEBIAN_SNORT_STATS_RCPT="root"
    DEBIAN_SNORT_STATS_THRESHOLD="1"
        

     

Install Oinkmaster and Update Snort Rules

Oinkmaster is a program used to update snorts detection rules. You will need to go to snort.org and sign up to receive an oink key; this lets your download rules.

  1. Install Oinkmaster
    #apt-get install oinkmaster
  2. Edit Oinkmaster Configuration File
    #nano /etc/oinkmaster.conf 
  3. Add Rule URLS to Oinkmaster Configuration File
    url = http://www.snort.org/pub-bin/oinkmaster.cgi/<your oink code>/
    snortrules-snapshot-2.7.tar.gz
    url = http://www.snort.org/pub-bin/oinkmaster.cgi/<your oink code>/
    Community-Rules-CURRENT.tar.gz
  4. Dowload and Install Rules
    #oinkmaster -C /etc/oinkmaster.conf -o /etc/snort/rules
    #perl /usr/share/ossim/scripts/create_sidmap.pl /etc/snort/rules/

Fix RRD Graphs

Ossim includes RRD for graphing values, I noticed that a table used for this purpose was missing. Here is how to fix that.

  1. Go to your home folder and create a text file holding code for the missing table
    #cd ~
    #nano snort.event_stats.sql
  2. Paste the following into the newly created file
    CREATE TABLE `event_stats` (
    `timestamp` datetime NOT NULL,
    `sensors` int(10) unsigned NOT NULL,
    `sensors_total` int(10) unsigned NOT NULL,
    `uniq_events` int(10) unsigned NOT NULL,
    `categories` int(10) unsigned NOT NULL,
    `total_events` int(10) unsigned NOT NULL,
    `src_ips` int(10) unsigned NOT NULL,
    `dst_ips` int(10) unsigned NOT NULL,
    `uniq_ip_links` int(10) unsigned NOT NULL,
    `source_ports` int(10) unsigned NOT NULL,
    `dest_ports` int(10) unsigned NOT NULL,
    `source_ports_udp` int(10) unsigned NOT NULL,
    `source_ports_tcp` int(10) unsigned NOT NULL,
    `dest_ports_udp` int(10) unsigned NOT NULL,
    `dest_ports_tcp` int(10) unsigned NOT NULL,
    `tcp_events` int(10) unsigned NOT NULL,
    `udp_events` int(10) unsigned NOT NULL,
    `icmp_events` int(10) unsigned NOT NULL,
    `portscan_events` int(10) unsigned NOT NULL,
    PRIMARY KEY (`timestamp`),
    KEY `sensors_idx` (`sensors`),
    KEY `sensors_total_idx` (`sensors_total`),
    KEY `uniq_events_idx` (`uniq_events`),
    KEY `categories_idx` (`categories`),
    KEY `total_events_idx` (`total_events`),
    KEY `src_ips_idx` (`src_ips`),
    KEY `dst_ips_idx` (`dst_ips`),
    KEY `uniq_ip_links_idx` (`uniq_ip_links`),
    KEY `source_ports_idx` (`source_ports`),
    KEY `dest_ports_idx` (`dest_ports`),
    KEY `source_ports_udp_idx` (`source_ports_udp`),
    KEY `source_ports_tcp_idx` (`source_ports_tcp`),
    KEY `dest_ports_udp_idx` (`dest_ports_udp`),
    KEY `dest_ports_tcp_idx` (`dest_ports_tcp`),
    KEY `tcp_events_idx` (`tcp_events`),
    KEY `udp_events_idx` (`udp_events`),
    KEY `icmp_events_idx` (`icmp_events`),
    KEY `portscan_events_idx` (`portscan_events`)
    ) ENGINE=MyISAM DEFAULT CHARSET=latin1;

    Save the file and run the following command to create the eventstats table in the snort database.

    #cat snort.event_stats.sql | mysql -p snort

Update Other Components

Some of the components that come with OSSIM are old, but you can easily update them by following instructions found in the OSSIM forums.

  1. Upgrade Nmap: see here.
  2. Upgrade Nessus: see here.
  3. Update Nessus rules using Alienvault’s free feed: see here.

It would be wise to reboot after installing all of the above. Afterwards, go to the web interface of your OSSIM server http://<ossim ip address>. You should see something like this:

Define your LAN(s) and run some nessus scans. Happy monitoring.

Cheers.

New Year’s Resolutions

  1. Drink less beer
  2. Drink more water
  3. Go to bed earlier
  4. Wake up earlier
  5. Switch to morning workouts
  6. Finish CCNA certification
  7. Get MCTS certified
  8. See Dave Matthews Band
  9. Learn to play the guitar
  10. Make time for a real relationship
  11. Learn to snow ski

Metallica can still play

I managed to snag some tickets to Metallica this past weekend. Definately worth it. Granted, I don't particularly care for their newer releases. Fortunately, these guys played a lot of pre-black album stuff. It was worth it to hear One and Fade to Black. 

Fade to Black

Oh, and they like to spit a lot. 

Metallica

 These guys have a lot of props. They brought at least 6 or 7 semi-trailers worth of equipment. There were huge moving coffins filled with lights, cameras and lasers constantly coming down and going up in ceiling.

Metallica likes its coffins

 

Apparently, most of what they brought was combustable.

One

 They are very meticulous with their sound.  Lars is constantly running off the stage between sets to adjust levels.

Metallica

If you don't mind when your ears ringing for 3 days afterwards, check them out on tour.

 

Upgrading Windows 2003 Domain to 2008 Domain

Fortunately, my company is enrolled in a 6 year microsoft subscription. This probably sounded appealing when Microsoft partners gave their speech (a year before they had a dedicated IT professional). I admit that it is nice to have a server license with the ability to upgrade to the current version at the time of my choosing. A company definitely saves money over time. Anyheuw, thats what I did. I have a single forrest with one domain.

The domain I inherited was originally a windows 2000 domain. This became obvious after examining the selected options and functional levels. Apart from these inconveniences, upgrading from 2003 DCs to 2008 DCs was a snap. Important: You will need to raise the forrest and domain functional levels to 2k3 native to do the upgrade, unless thats where your forrest/domain is now. Windows 2k3 forrest/domains are in a Win2k compatibility functional level by default. Make sure to give your functional raise some time to propagate.  

After my forrest and domain functional levels were raised, I moved all 5 fsmo roles to a single “virgin” 2003 DC that was born for this adventure. This served to eliminate possible third-party/previous install conflicts.  Domain controllers seem to become a slave of small third-party semi-important services on a small Windows domain = too many on mine to think about fighting with.

Next I did a series of health tests on my Active Directory. I used Windows 2003 support tools to out put a series of tests’ results to text files and studied them a bit. The Event Viewer is also helpful when improving AD health.  Upgrading to a new AD schema wont improve existing AD problems, it will only complicate them. At the very least, make sure that dcdiag and netdiag tests all pass and fix the event logged errors on your DCs.

After AD was healthy and all controllers were synced up, I put up a new 2k8 server and joined it to the domain. From my newly created win2k3 box, I prepped my domain for 2k8 

  1. Insert Win2k8 DVD (either x64 or x86 = depending on what the “virgin” 2k3 domain controller is) and copy the adprep folder (on the DVD its \sources\adprepto the HD
  2. Upgrade AD definitions in the forrest: cmd to the root of the folder you copied -> adprep.exe /forrestprep
  3. Allow the domain to take advantage of what the forrest has to offer: stay in cmd -> adprep.exe /domainprep
  4. Upgrade the group policy to match the newly updated AD schema of the domain: adprep.exe /domainprep /gpprep
  5. Optionally allow for read only domain controllers: adprep.exe /rodcprep
From my 2k8 box, I ran dcpromo (command line) and followed the wizard. Make sure to select advanced options and add to the existing forrest & domain. Good to go. From there created another 2k8 (x64) controller and demoted/removed my old 2k3 controllers. Hint: If you are demoting, turn off Trend Micro on the DCs if thats what you use for AntiVirus.
All in all, I think 2k8 runs a better domain controller than 2k3. Its definitely less boring to push buttons in = more intuitive. Memory usage is greater in 2k8 than that in 2k3 to handle Windows Domain roles. However, its memory usage does improve performance…or maybe I am fooled by a more resonsive GUI over terminal services. In any case, I am a fan of the greater control over my domain via the updated Group Policy/DNS options; just make sure you raise your forrest & domain functional levels to 2k8 reap all the benefits. (P.S. This means you cannot have domain controllers running anything less than 2k8)
Now its time to replace the 2k3 exchange/windows box with a 2k7 exchange/2k8 windows (x64) box. The enhanced, built-in spam filtering of Exchange 2k7 looks good to me right now. Recently, my users have been getting mail saying things such as “I have kidnapped your baby, give me money to see her again” and “click here” to get your gift card. As much as I am a fan of gift cards, I care too much for the children to allow the kidnapping/ransom notes to continue.
Cheers.