Category Archives: vCenter

My vCenter C: drive ran out of space

I came across something interesting today. The 160 GB C: drive on my vCenter Server ran out of space today…rather embarrassing. The first thing I checked is why in the heck my alerts didn’t go off…ok..problem fixed. After a couple of Google searches I came across and interesting VMware KB. Apparently there is a bug in the vCenter 5.5 upgrade that enables debug logging on the VMware Syslog Collector service and logs to C:\ProgramData\VMware\VMware Syslog Collector\logs\debug.log. In this case, my debug log was 62GB. The fix is rather simple. Stop the ‘VMware Syslog Collector’ service and edit the C:\ProgramData\VMware\VMware Syslog Collector\vmconfig-syslog.xml file (make a copy of it first) and change the following:

<debug> <level>1</level> </debug>

to

<debug> <level>0</level> </debug>

 

Remove the debug.log file and start the service again.

 

After upgrading to vCenter Server 5.5, the debug.log file of syslog collector is growing without limit (2094175)

KB: 2094175

  • Updated: Mar 3, 2015
  • Categories:
    Troubleshooting
  • Languages:
    English
  • Product(s):
    VMware vCenter Server
  • Product Version(s):
    VMware vCenter Server 5.5.x

 

Symptoms

  • After upgrading to vCenter Server 5.5, the debug.log file of syslog collector is growing without limit
  • C:\ProgramData\VMware\VMware Syslog Collector\logs\debug.log continues to grow without being rotated

Resolution

This is a known issue affecting VMware Syslog Collector 5.5.

 

Currently, there is no resolution.

To work around this issue:

  1. Stop the VMware Syslog Collector service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).
  2. On the server running the VMware Syslog Collector service, navigate to C:\ProgramData\VMware\VMware Syslog Collector and save a copy of vmconfig-syslog.xml.
  3. In a text editor, open vmconfig-syslog.xml and modify  from:

    <debug> <level>1</level> </debug>

    to

    <debug> <level>0</level> </debug>

  4. Start the VMware Syslog Collector service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).

 

-Aaron

vCenter Server 6.0 New Features

Today Announces the release of VMware’vCenter largest release in VMware History. Note. These facts are pre release, thus I will update any changes if necessary, All Facts are direct From VMware.

vCenter Server Features – Enhanced Capabilities

Metric
Windows
Appliance
Hosts per VC
1,000
1,000
Powered-On VMs per VC
10,000
10,000
Hosts per Cluster
64
64
VMs per Cluster
8,000
8,000
Linked Mode

Platform Services Controller 

Platform Services Controller includes takes it beyond just Single Sign-On. It groups:

  • Single Sign-On (SSO)
  • Licensing
  • Certificate Authority
Two Deployment Models:
Embedded
vCenter Server and Platform Services Controller in one virtual machine
– Recommended for small deployments where there is less then two SSO integrated solutions
Centralized
vCenter Server and Platform Services Controller in their own virtual machines
– Recommended for most deployments where there are two or more SSO integrated solutions

Linked Mode Comparison

vSphere 5.5
vSphere 6.0
Windows
Yes
Yes
Appliance
No
Yes
Single Inventory View
Yes
Yes
Single Inventory Search
Yes
Yes
Replication Technology
Microsoft ADAM
Native
Roles & Permissions
Yes
Yes
Licenses
Yes
Yes
Policies
No
Yes
Tags
No
Yes

Certificate Lifecycle Management for vCenter and ESXi

VMware Certificate Authority (VMCA)

Provisions each ESXi host, each vCenter Server and vCenter Server service with certificates that are signed by VMCA

VMware Endpoint Certificate Service (VECS)

Stores and Certs and Private Keys for vCenter Services

vCenter Server 6.0 – VMCA

Root CA

  • During installation, VMCA automatically creates a self-signed certificate
  • This is a CA certificate, capable of issuing other certificates
  • All solutions and endpoint certificates are created (and trusted) from this self-signed CA certificate

Issuer CA

  • Can replace the default self-signed CA certificate created during installation
  • Requires a CSR issued from VMCA to be used in an Enterprise/Commercial CA to generate a new Issuing Certificate
  • Requires replacement of all issued default certificates after implementation

Certificate Replacement Options for vCenter Server

VMCA Default

  • Default installed certificates
  • Self-signed VMCA CA certificate as Root
  • Possible to regenerate these on demand easily

VMCA Enterprise

  • Replace VMCA CA certificates with a new CA certificate from the Enterprise PKI
  • On removal of the old VMCA CA certificate, all old certificates must be regenerated

Custom

  • Disable VMCA as CA
  • Provision custom leaf certificates for each solution, user and endpoint
  • More complicated, for highly security conscious customers

Cross vSwitch vMotion

  • Transparent operation to the guest OS
  • Works across different types of virtual switches
  • vSS to vSS
  • vSS to vDS
  • vDS to vDS
  • Requires L2 network connectivity
  • Does not change the IP of the VM
  • Transfers vDS port metadata

Cross vCenter vMotion

  • Simultaneously changes
  • Compute
  • Storage
  • Network
  • vCenter
  • vMotion without shared storage
  • Increased scale
  • Pool resources across vCenter servers
  • Targeted topologies
  • Local
  • Metro
  • Cross-continental

Requirements

  • vCenter 6.0 and greater
  • SSO Domain
  • Same SSO domain to use the UI
  • Different SSO domain possible if using API
  • 250 Mbps network bandwidth per vMotion operation
  • L2 network connectivity on VM portgroups
  • IP addresses are not updated

Features

  • VM UUID maintained across vCenter server instances
  • Not the same as MoRef or BIOS UUID
  • Data Preservation
  • Events, Alarms, Tasks History
  • HA/DRS Settings
  • Affinity/Anti-Affinity Rules
  • Automation level
  • Start-up priority
  • Host isolation response
  • VM Resource Settings
  • Shares
  • Reservations
  • Limits
  • MAC Address of virtual NIC
  • MAC Addresses preserved across vCenters
  • Always unique within a vCenter
  • Not reused when VM leaves vCenter

Long Distance vMotion 

  • Cross-continental distances – up to 100ms RTTs
  • Maintain standard vMotion guarantees
  • Does not require VVOLs
  • Use Cases:
  • Permanent migrations
  • Disaster avoidance
  • Multi-site load balancing
  • Follow the sun

Increased vMotion Network Flexibility

 

  • vMotion network will cross L3 boundaries
  • vMotion can now use it’s own TCP/IP stack

Content Library Overview

  • Simple content management
  • VM templates
  • vApps
  • ISO images
  • Scripts
  • Store and manage content
  • One central location to manage all content
  • Beyond templates within vCenter
  • Support for other file types
  • Share content
  • Store once, share many times
  • Publish/Subscribe
  • vCenter  -> vCenter
  • vCloud Director -> vCenter
  • Consume content
  • Deploy templates to a host or a cluster

Client Overview and Web client Changes

 Client Comparison

vSphere Client

  •  It’s still here
  • Direct Access to hosts
  • VUM remediation
  • New features in vSphere 5.1 and newer are only available in the web client
  • Added support for virtual hardware versions 10 and 11 *read only*

vSphere Web Client

  • Performance
  • Improved login time
  • Faster right click menu load
  • Faster performance charts

Usability

  • Recent Tasks moved to bottom
  • Flattened right click menus
  • Deep lateral linking
  • Major Performance Improvements

UI

  • Screen by screen code optimization
  • Login now 13x faster
  • Right click menu now 4x faster
  • Most tasks end to end are 50+% faster
  • Performance charts
  • Charts are available and usable in less then half the time
  • VMRC integration
  • Advanced virtual machine operations
  • Usability Improvements
  • Can get anywhere in one click
  • Right click menu has been flattened
  • Recent tasks are back at the bottom
  • Dockable UI

vCenter Cluster Support



vCenter server is now supported to be ran in a Microsoft Cluster.




That’s all of the changes we were presented with from VMware. What a ton of changes, I will dig into these more soon.

 

 

Update: a post vSphere 6 – Clarifying the misinformation has been posted to clairify any changes that have or will happen between beta and this post. I did my best to validate that my information is correct.

 

Roger L

Interesting KB for After upgrading to vCenter Server 5.0 Update 1

 

I recently upgrade from vCenter 5.0 to 5.0 U1, and ran into this issue.

 

I saw the following problems.

 

 

I see upgrade vCenter Agents on cluster hosts in recent tasks.

And after a services restart, It would disconnect all VMware Hosts.

 

The following KB Addresses this.

 

I thought it would be a good read for those of you that haven’t pulled the trigger for a U1 update yet.

 

After upgrading to vCenter Server 5.0 Update 1, vCenter Server attempts to upgrade the vpxa agent and all hosts are marked as disconnected after a restart

 

 

KB: 2016287

    * Updated: Apr 2, 2012
    * Categories:
      Troubleshooting
    * Product Family:

    * Product(s):
      VMware vCenter Server

    * Product Version(s):
      VMware vCenter Server 5.0.x

 

Symptoms

After upgrading to vCenter Server 5.0 Update 1, you experience these symptoms:

    * vCenter Server tries to upgrade the vpxa agent and all hosts are marked as disconnected
    * In vCenter Server, you see the Upgrade vCenter Agents on cluster hosts task in progress, but the task never completes
    * This issue may occur if the upgrade of vpxa agents was set to manual in a previous upgrade of vCenter Server
    * In the vpxd.log file, you see entries similar to:

vpxd-14370.log:2012-03-21T08:43:19.447Z [fusion_builder_container hundred_percent=”yes” overflow=”visible”][fusion_builder_row][fusion_builder_column type=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”no” center_content=”no” min_height=”none”][03512 info ‘Default’ opID=SWI-b1839753] [VpxdHostUpgrader] Preinstalled bundle found: not installing
vpxd-14370.log:2012-03-21T08:43:19.463Z [03764 info ‘Default’ opID=SWI-758aecea] [VpxdHostUpgrader] Preinstalled bundle found: not installing
vpxd-14370.log:2012-03-21T08:43:19.541Z [03740 info ‘Default’ opID=task-1159161-e85e8e4e] [VpxLRO] — BEGIN task-1159161 — domain-c10910 — ClusterUpgrade.UpgradeAgent —
vpxd-14370.log:2012-03-21T08:43:19.666Z [03740 info ‘DAS’ opID=task-1159161-e85e8e4e] [VpxdDasUpgradeLRO] Waiting for hosts to finish upgrading
vpxd-14370.log:2012-03-21T08:43:19.666Z [03740 info ‘DAS’ opID=task-1159161-e85e8e4e] [VpxdDasUpgradeLRO] Number of hosts upgrading: 4
vpxd-14370.log:2012-03-21T08:43:19.682Z [03696 info ‘Default’ opID=task-1159163-2bda221f] [VpxLRO] — BEGIN task-1159163 — domain-c7 — ClusterUpgrade.UpgradeAgent —
vpxd-14370.log:2012-03-21T08:43:19.682Z [03736 info ‘Default’ opID=task-1159162-b0d08563] [VpxLRO] — BEGIN task-1159162 — domain-c347 — ClusterUpgrade.UpgradeAgent —
vpxd-14370.log:2012-03-21T08:43:19.682Z [03736 info ‘DAS’ opID=task-1159162-b0d08563] [VpxdDasUpgradeLRO] Waiting for hosts to finish upgrading
vpxd-14370.log:2012-03-21T08:43:19.682Z [03736 info ‘DAS’ opID=task-1159162-b0d08563] [VpxdDasUpgradeLRO] Number of hosts upgrading: 2
vpxd-14370.log:2012-03-21T08:43:19.682Z [03720 info ‘Default’ opID=task-1159164-72891c06] [VpxLRO] — BEGIN task-1159164 — domain-c71 — ClusterUpgrade.UpgradeAgent —
vpxd-14370.log:2012-03-21T08:43:19.682Z [03696 info ‘DAS’ opID=task-1159163-2bda221f] [VpxdDasUpgradeLRO] Waiting for hosts to finish upgrading
vpxd-14370.log:2012-03-21T08:43:19.682Z [03696 info ‘DAS’ opID=task-1159163-2bda221f]
Cause
This issue occurs if the autoUpgradeAgents option in vCenter Server advanced settings is set to false and vCenter Server incorrectly tries to upgrade the vpxa agent a second time on the ESX/ESXi hosts
Resolution
To workaround this issue, change the autoUpgradeAgents option to true.
 
To change the autoUpgradeAgents option to true:

   1. Click Administration > vCenter Server Setting > Advanced Settings.
   2. Look for the AgentUpgrade.autoUpgradeAgents option and change this setting from false to true.
   3. Restart the VMware VirtualCenter Server service.
   4. Manually reconnect all ESX hosts.

      The Upgrade vCenter Agents on cluster hosts task should now complete.

I thought it was a good to know KB.

 

 

 

Roger Lund

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

VI / vCenter cannot connect to Host, Service mgmt-vmware restart may not restart hostd

Yesterday, I had a problem after I shutdown a lagcy San, that after I rescaned my HBA’s, ( which timed out), I lost connection to the host.

I tried restarting the normal services, in console,

# service vmware-vpxa restart

#service mgmt-vmware restart

but I would get,

Stopping VMware ESX Server Management services:
   VMware ESX Server Host Agent Watchdog                   [fusion_builder_container hundred_percent=”yes” overflow=”visible”][fusion_builder_row][fusion_builder_column type=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”no” center_content=”no” min_height=”none”][FAILED]

 

I found the following post , http://communities.vmware.com/message/1157237 and KB article, http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005566

 

Both which have you kill the hostd, process, then rename the pid, file. Here is the contents of the KB.

"

Service mgmt-vmware restart may not restart hostd

Symptoms
  • You are using the command  service mgmt-vmware restart but it does not finish restarting hostd.
  • The script gets stuck when stopping the service.
  • The SSH session to the ESX host becomes unresponsive.

  • hostd does not restart

Resolution

You must manually stop the stuck service and restart it.

To stop the service and restart it:

  1. Log in as root to the ESX host command-line via the physical console or via KVM connection.

  2. Navigate to the /var/run/vmware directory:
    # cd /var/run/vmware
  3. Run the following command to list the files vmware-hostd.PID and watchdog-hostd.PID:
    # ls -l vmware-hostd.PID watchdog-hostd.PID

  4. Determine the Process ID (PID) management service. View the contents of the vmware-hostd.PID file: 
    # cat vmware-hostd.PID
    For example:
    [[email protected]]# cat vmware-hostd.PID
    1191[[email protected]]#
  5. Use the resulting PID to kill the process.
    Caution: Use the kill -9 command with care. It kills the process of the supplied PID without exception or confirmation.
    # kill -9 <PID>
    In this example you run kill -9 1191.

  6. Delete the vmware-hostd.PID and watchdog-hostd.PID files:
    # rm vmware-hostd.PID watchdog-hostd.PID
  7. Restart the management service.
    # service mgmt-vmware restart”

This seemed to fix it, and without any downtime.

 

Roger L

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]