Home Storage
Category:

Storage

DatacenterStorageVirtualization

Finally bought a Drobo for home

by Philip Sellers
written by Philip Sellers

Last week, I found a deal I could not pass up.  B&H Photo has a deal on a Drobo for $299 though 6/30/2010.  If you’ve never heard of a Drobo, it is an external storage enclosure from Data Robotics that offers some enterprise-class, automated mirroring/striping for your data across multiple hard drives.  Data Robotics calls it Beyond-RAID because unlike a RAID set where drives should be the same size, their technology allows mix and match drive sizes and handles striping and leveling the data across whatever mix of SATA drives you buy.  If a drive fails, pull it and replace it and the device will rebuild.

I had been worried about losing my digital home movies.  That data is really too large to really push out to a backup service like Mozy and when I load new movies, its usually to the tune of 20 or 30GB at a time, which would take weeks to push up.  In addition to that data, I also have Movies and TV shows that we have purchased through iTunes.

Continue Reading
3 comments
0 FacebookTwitterPinterestLinkedinRedditEmail
DatacenterNetworkingStorageVirtualization

Monitoring challenges moving critical systems in virtual infrastructure

by Philip Sellers
written by Philip Sellers

Like many shops, we have finally attained buy-in from all our stakeholders for virtualization.  As a result, we’ve pushed more and more into our infrastructure.  And while VMware is the most datacenter ready solution for virtualization, it is not without its shortcomings — monitoring and visibility into the infrastructure being one of the biggest.

While we were first deploying VI3 and performing our consolidation, the primary focus was on the non-critical systems and moving them into the virtual infrastructure to get the best utilization of hardware.  Since completion of this phase, the next focus became moving some of our mission critical systems to VMware in order to establish disaster recovery for our non-clustered systems.  Disaster recovery through VMware is accomplished by 1) relocating the boot and data onto SAN storage which is replicated to our secondary data center and 2) by the ability to utilized VMware HA in the event of hardware failure to establish resiliency we do not have on a single-server, hardware deployment.

As we have expanded VMware’s role in our data center, new challenges have emerged.  First, when a network issue is occurring, we don’t have our traditional monitoring tools (like PRTG) in a position where they are able to alert for large changes in traffic.  In our physical environment, HP agents are run and PRTG is able to query against these systems with SNMP to retrieve information about traffic.  In the virtual environment, we don’t run these agents (because they are largely non-applicable since these are virtual boxes).  Our preferred way to monitor is through something that can look directly into the virtualization layer and retrieve information.

Continue Reading
1 comment
0 FacebookTwitterPinterestLinkedinRedditEmail
DatacenterStorageVirtualization

Path failures on ESX4 with HP storage

by Philip Sellers
written by Philip Sellers

Since we began upgrading our clusters to ESX4, we have been having strange “failed physical path” messages in our vmkernel logs.  I don’t normally post unless I know the solution to a problem, but in this case, I’ll make an exception.  Our deployment has been delayed and plauged by the storage issues that I mentioned in an earlier post.  Even though we have fixed our major problems, the following type errors have persisted.

Our errors look like this:

vmkernel: 19:18:05:07.991 cpu6:4284)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410005101140) to NMP device “naa.6001438005de88b70000a00002250000” failed on physical path “vmhba0:C0:T0:L12” H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
vmkernel: 19:18:05:07.991 cpu6:4284)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device “naa.6001438005de88b70000a00002250000” state in doubt; requested fast path state update…

After several cases with VMware and HP technical support, we are no closer to resolving the issues.  VMware support, for its part, has done a good job of telling us what ESX is reacting to and seeing.  HP support, on the other hand, has been circling around the problem but has made little progress in diagnosing the issue.  We have had an ongoing case for several months and our primary EVA resource at HP has continually examined the EVAperf information and SSSU output that we have sent to HP for analysis.  Those have turned up nothing, and yet the messages continue from VMware.

The errors in the log make sense to me – we are losing a path to a data disk (sometimes even a boot-from-SAN ESX OS disk!) – but why HP cannot see anything in our Brocade switches or within the EVA is beyond me.   Our ESX hosts, whether blade or rack-mounted hardware, are seeing the problems across the board.  The one cluster we waited to upgrade never saw the issues in ESX3.5, but sees them now in ESX4.  And perhaps it is a VMware issue that is just too sensitive in monitoring its storage, but I suspect its something else.   The messages don’t seem to affect operation on the hosts, but it certainly makes investigating problems difficult when trying to determine what is a real problem versus just another failed path message.  Anyone else seeing this?

4 comments
0 FacebookTwitterPinterestLinkedinRedditEmail
DatacenterStorageVirtualization

Best practices for VMware ESX4 with HP EVA storage

by Philip Sellers
written by Philip Sellers

(For PowerCLI commands to accomplish the same best practices, see this post).

HP provided us with the best practices document for ESX4 connected to an HP EVA array.  There is a major change in ESX4.  For the first time, ESX is ALUA (Asymmetric Logical Unit Access) aware.  (See this post on Yellow Bricks for more detail about ALUA.)  ALUA allows the array and ESX to determine the optimal path — the path to the managing controller’s ports in the EVA’s case — and use those optimal paths until one isn’t available.  This is important because it prevents flip-flopping on the ESX host.

In previous version of ESX, the desired storage setting was fixed path for the EVA.  In our case, we simultaneously presented the ESX3.5 and ESX4 hosts to the same LUNs, meaning some were fixed and some were set to the ESX4 default, which was MRU.  This caused problems.  After initial issues, we backed away and presented one LUN at a time, performed our VMotions and then unpresented the LUN from the old cluster.  This prevented any flapping issues between controllers.

Continue Reading
3 comments
0 FacebookTwitterPinterestLinkedinRedditEmail
DatacenterNewsStorage

Collecting diagnostic information for vCenter Server 4.0 , Missing Files, Internal Error, Log Export Failed?!

by Roger Lund
written by Roger Lund

I found my self needing to generate diagnostic information for vCenter Server 4.0 U1. I have done this in the past, using the following steps.

“


  1. Log in to VirtualCenter with the vSphere Client, as an administrator or log in directly to an ESX Server with the root user.

  2. Click File > Export > Export System Logs.

  3. On the Save Export System Logs Window, choose the host(s) to collect diagnostic data from and whether to include data from VMware Virtual Center.


    Note: If you are directly connected to an ESX Server skip this step as the Diagnostic Data Component of the Window is not displayed.

Click Browse and specify a location to save the diagnostic data to. The following shows the Export System Logs dialog:

Note: Selecting the Include information from vCenter Server and vSphere Client option includes additional non-ESX related logs.

 4. Click OK to start the generation of the logs.

* taken from http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=653

“


And I have always had good luck, zipped the files and away I went uploading them to FTP.

But today, I got a message saying, Missing files. I then tried this direct from the host, also using VI client, and It gave me internal error.

Hmm, a google search didn’t get any hits.

I then reviewed the KB’s on how to export logs.


Collecting diagnostic information for VMware vCenter Server 4.0

KB Article: 1011641

ok, this looks, like the steps I was using. 

Collecting diagnostic information for VMware ESX Server 2.5, 3.0, 3.5 and 4.0

KB Article: 653

“


Obtaining Diagnostic Information from the ESX service console

Diagnostic information can be obtained directly from any version of ESX host. Information cannot be obtained from ESXi hosts in this way.

To collect diagnostic data at the ESX command prompt:

  1. Log on to the service console as root.

  2. Confirm the version of VMware ESX Server you are running.

    • On ESX Server 2.x and 3.x hosts, determine the base build number with the following command:


      [fusion_builder_container hundred_percent="yes" overflow="visible"][fusion_builder_row][fusion_builder_column type="1_1" background_position="left top" background_color="" border_size="" border_color="" border_style="solid" spacing="yes" background_image="" background_repeat="no-repeat" padding="" margin_top="0px" margin_bottom="0px" class="" id="" animation_type="" animation_speed="0.3" animation_direction="left" hide_on_mobile="no" center_content="no" min_height="none"][root@esxhost]# vmware -v 

    • On ESX 4.x hosts, determine the system level with the following command:


      [root@esxhost]# vmware -l

    • Run the following command to change to the directory where you want the output to appear. For example:

      [root@esxhost]# cd /tmp


      Note: If you run the script in /usr/bin, the output appears in that directory and remains there until you delete it.

    Run the following command to execute the script:

    [root@esxhost]# /usr/bin/vm-support


    You do not need to power off your virtual machines before running this script.

    For ESX Server performance issues, VMware Technical Support might ask you to collect performance snapshots using the the -s and -S switches. For more information, see Collecting performance snapshots using vm-support (1967) .


    When the script finishes, it generates a file with named [email protected]’

    New directions? OK, let’s try that.

    # cd / 
    # cd /temp
    # /usr/bin/vm-support

    It worked!

    But, why?

    Let’s do some investigative work to see what is going on.

    First let’s do a df -h to see free space, this also display’s your partitions.


     Hmm, / seems really full, I wonder why?

    How do I tell what is taking that room?

    Thanks to Jason Boche via, twitter, I found that du -h would show used files per directory.

    See KB 1016511 ESX 4 may generate a purple diagnostics screen or kernel panic after an upgrade due to an insufficient disk space

    “If you are unable to determine where the space is being used, you can run the following command:

    [root@server]# find / -size +10240000c -exec du -h {} ; | less

    This command provides a list of all files that are larger than 10MB in size. 

    After you have determined what is taking up the free space you can use the rm command to free up space by deleting files.


    Warning: When you delete a file there is no way to recover it, so use caution when deleting anything.

    The following are a list of files that are safe to delete:

    • Old vm-support logs, usually under /usr/lib/vmware/hostd/docroot/downloads/

    • Virtual machines that are not needed and were copied to service console for storage

    • ISO files that were copied to the system. In many cases under the /vmimages folder

    Note: Do not delete the VMware Tools ISO files.”

     I had to at a 0 to 10240000c, to narrow down the results. I also found that q exists the command.

     I found I had several large files in /usr/lib/vmware/hostd/docroot/downloads/


     I didn’t get a screen of each size, but they totaled over 2 GB.

    I then looked to see the correct method to delete these files, as above it says it is save to deleted files in this directory.

    I found KB 1008970  After upgrading ESX, esxcfg-boot fails with a More than one VMkernel found error“

    Run the following command to delete these directories: 


    # rm -rf /var/lib/vmware/hostd/docroot/downloads/vm-support-*


    or


    # rm -rf /usr/lib/vmware/hostd/docroot/downloads/vm-support-*

    “

    Or, I found that you can use something like WinSCP or Veeam’s FactSCP to browse the directory, and delete the files.


     let’s try df -h again.

    Much better!

    I hope this helps someone else in this situation.


    As I was doing this process, I thought to my self that I could be a good idea to create a separate /usr partition, then you wouldn’t have / fill up from support files.

    Could it also be a good idea to make /root a separate partition as well? After all, some large files can be stored there.


     This lessen also reminds us all to keep a eye on our local disk, even if you do not have a VMFS datastore on it.

    Thoughts on the partitioning, or anything else?

    Roger L.

    [/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

    0 comments
    0 FacebookTwitterPinterestLinkedinRedditEmail
    DatacenterNewsStorage

    Running ESX Host kit 5.1 for NetApp on vSphere , DNS error’s

    by Roger Lund
    written by Roger Lund

    I had a ESX host that would give me the following error when I ran my config mpath.


    #config_mpath –primary –loadbalance –policy fixed




    WARNING:
       Your DNS is improperly configured.
       The following storage system hostname(s) could not be resolved:
          scoa-netac1
          scoa-netac2


       After the DNS issues are resolved, you must properly
       configure the preferred paths for NetApp storage systems
       by running the following command:
          /opt/netapp/santools/config_mpath –primary –loadbalance –persistent


    I checked my dns, and my FQDN given was in uppercase , therefore, I changed it and tried again.

    ——

    WARNING:
       Your DNS is improperly configured.
       The following storage system hostname(s) could not be resolved:
          scoa-netac1
          scoa-netac2
       After the DNS issues are resolved, you must properly
       configure the preferred paths for NetApp storage systems
       by running the following command:
          /opt/netapp/santools/config_mpath –primary –loadbalance –persistent
    —–
    Now what?
    The next step was to manually check my configuration files.
    #nano /etc/resolv.conf
    Showed: ( note, I changed my below info)
    “
    nameserver 1x.220.0.x
    nameserver 1x.220.0.x
    search domain.com
    “
    Which is correct.
    So then I did.
    #nano /etc/hosts
    “
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1       localhost.localdomain localhost
    1x.220.0.x     SCOA-VIRTUAL2.domain.com
    1x.220.0.x   scoa-netac1.domain.com scoa-netac1
    1x.220.0.x   scoa-netac2.domain.com scoa-netac2
    ::1             localhost.localdomain localhost
    “
    Ah, Ha! We changed netac1 and netac2’s IP!
    I commented out those lines, and yahoooo, it worked!
    Roger L.


    0 comments
    0 FacebookTwitterPinterestLinkedinRedditEmail
    AppleDatacenterHome TechStorage

    Apple’s Mac Mini server is popular, what could be next?

    by Philip Sellers
    written by Philip Sellers

    When Apple last refreshed the Mac Mini line, it introduced a new version with Mac OS X Server preloaded that also dropped its optical drive in lieu of dual hard drives. This new configuration is quite popular according to a recent post on AppleInsider.  That leads me to wonder and speculate about what could be next? I’ve speculated about a home server before…

    Mac OS X Server has been incrementally growing in features and capabilities with each major release.  Its market penetration and target has largely been towards Mac heavy businesses.  It has packaged several new products in the last two releases to help Mac organizations reduce their dependence on Microsoft solutions – new products like iCal and Address Book server as well as a mail server, all of which are based heavily on standards which is great for interoperability across all client platforms.

    But, with the introduction of the Mac Mini server, Apple has tilted the OS X Server offering more towards small businesses (who might not purchase a Mac Pro or X-Serve to run OS X Server).  It also could be used for some home users — could this be the first, quiet step towards the home server for Apple?   I don’t have to answer and this is purely speculation, but let me continue.

    Continue Reading
    1 comment
    0 FacebookTwitterPinterestLinkedinRedditEmail
    DatacenterStorageVirtualization

    Carefully planning ESX4 and HP Storageworks EVA

    by Philip Sellers
    written by Philip Sellers

    As in my post about Lessons Learned on ESX4 rollout, we had a pretty serious hiccup with our storage and the ESX systems in December while trying to bring up our ESX4 environment.  The primary trouble uncovered was what I’ll call “controller ping-pong”.

    An EVA normally has two (maybe more, I’m not primarily a storage guy) controllers and those handle all the requests received through the SAN.  For every LUN, one controller is its master.  Both controllers can handle requests for the LUN, but only one actually handles the access.  If the controller on fabric A is the primary but the controller on fabric B is getting more requests, eventually the EVA swaps control for the LUN to fabric B — wherever the majority of requests are coming.

    This behavior would only become a problem if you had hosts configured to access the LUN on different fabrics.  ESX4 is ALUA (asymmetric logical unit access) aware, meaning it should automatically determine the optimal path and in the case of an EVA.  The EVA, I’m told by HP support, is supposed to respond an ALUA request for the optimal path by responding with the controller that is the master over the LUN.

    If you, like us, have an ESX 3.5 cluster with preferred paths setup, you should proceed with caution.  The ALUA information isn’t apparently shared between clusters.  And if your clusters get different optimal paths, you could end up with controller ping-pong as requests are sent down both fabrics and the volume changes between the two, resulting in more on Fabric A followed by more on Fabric B — forcing the controller to switch masters.

    So, while in a migratory state, I think my safest route is to configure the ESX4 hosts to use a preferred path like the ESX3.5 cluster nodes.  I hate to move from the default ESX configuration and this isn’t an official recommendation from HP support, but it certainly makes the most sense to define the paths being used (except in a failure).

    I post this because I feel like there have to be other HP Storageworks customers who have the same situation or have experienced something similar.  I would love to hear from you…

    3 comments
    0 FacebookTwitterPinterestLinkedinRedditEmail
    @2019 - All Right Reserved. Designed and Developed by PenciDesign