9/29/2014

USB3.0, Sustaining 88 megabytes/sec transfer rate


I have an ASUS P8Z68-V Gen3 Motherboard with USB3.0 ASMedia XHCI Host controllers.


Until I installed the [optional] ASUS "Boost" BOT Storage Driver, an attached USB Hard drive would drop to 10 MB/s after a few seconds of sustained transfer. After the "Boost" driver is installed it regularly "sustains" 88 MB/s transfer rates. This partially depends upon the capabilities of the device attached to the host controller.


ASUS describes their alternative storage driver as supporting the latest UAS Protocol (UASP) which is a workaround for the USB 2.0 BOT (Bulk Only Transfer) protocol originally designed for USB 2.0 "High Speeds". Theoretically 450 Mb/s divide that by 10 and multiply by 0.8 is practically 36 MB/s but most share the USB bus with other devices and the host controller sits on a PCI bus which allocates a reduced amount of bandwidth.. typically 15% so that comes out to about 5.4 MB/s

So obtaining 10 MB/s on a dedicated USB bus with no other devices is actually quite remarkable.

However using the "Boost" Turbo mode for the USB 3.0 BOT driver consistently returns 88 MB/s for me, and is quite worth the effort to figure all this out.

The details are available online at this URL

How ASUS USB 3.0 Boost UASP Mode accelerates transfer speed

Essentially it appears they are "streaming" the data in a deeper queue before confirming a transfer, taking advantage of greater reliability over the USB 3.0 bus, less retransmits become more efficient and this removes some of the overhead. For network engineers this is the equivalent of using "Jumbo Packets".

Additionally they "package" or bundle the transfers in some manner in "Turbo" mode to increase the speeds further. For the compression specialist think on the fly compression on the device and decompression on the host controller.

It all requires USB 3.0 hub hardware that supports the new modes,  an operating system driver that supports the new modes and a USB 3.0 device connected to the host controller root hub, that at least supports the alternative UASP protocol. The different choices being, run in "Normal" USB 2.0 BOT mode, run in "Enhanced" USAP BOT mode, or run in "Turbo" Boost BOT mode. (the knight rider would be impressed..)


What was missing was the new ASUS [host controller] "root hub" software driver for the operating system I was running. [ ASUS USB 3.0 Boost Storage Driver ]


This had to be downloaded from the ASUS website for my motherboard and operating system:


Installed

And rebooted the new "alternative" USB 3.0 BOT storage driver ["Boost"] is detected and installed, as well as the "normal" USB 2.0 BOT storage driver, there is no need to remove and reinstall the host controller drivers. If you do be aware that the host controller drivers that came on the DVD with your motherboard or system are probably a better "fit" than the latest ones available from the ASUS website or Microsoft driver update. The simple fact is once a batch of motherboards leaves the factory most driver development stops and both of these alternative sources of drivers are not rigourously tested with your particular motherboard. Its sad but true. So your best bet is to stick with host controller drivers you know work, at least until the "Boost" driver is working.


The rest is pretty much automatic, the new driver will detect if UASP or Turbo modes are possible and put them into effect when a compatible USB device is attached to the host controller root hub.

This motherboard has an additional system tray tool for interfacing with the BIOS and can also interact with hardware to show more information than is normally available through the operating system.

In this case it interacts with the USB 3.0 host controller driver to indicate the mode in which it is communicating with my Western Digital Passport USB 3.0 external hard drive.






9/11/2014

HP SIM, WBEMTest Indications using WMIC

Its rather unintuitive how to use WMIC to send a Test Indication to HP SIM, here's how to do it.

C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem where Caption="afs00-2.agnet.tamu.edu" CALL  SendTestIndication TestIndicationID=101

First you need to be aware that Methods on a Class can be [Static] or [Non-Static], a static method can be called at the same time as Instantiating (aka 'Creating') a New Instance of a Class. A non-static method can be called only an "Existing" Instance of a Class.

To send indications from a managed server using the HP SIM WMI Providers, they must be installed and running.

The HPWinComputerSystemProv WBEM/WMI Provider, comes with a Class called HP_WinComputerSystem which creates a New Instance on startup and has a [Non-Static] Method called SendTestIndication.

SendTestIndication takes exactly one [In ] Param TestIndicationID of Type CIM_UINT32 which maps through VT_I4 to WMI (uint32) -- basically "its a number, not a string" but even if you surround it in quotes when using wmic.exe, it will still treat it as a number and you will not have a type problem.

Ok, works - TestIndicationID=101
Ok, works - TestIndicationID="101"

As a concrete example you can "walk" the CIM using WMIC to Create a brand new "shiney" copy of any Object based on the HP_WinComputerSystem Class definition, and retrieve (all) of its default Properties and Values, as follows:

C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem  GET /all /format:list

And this brand new Object instance has a Method on it, we can "ask" how to use it, as follows:
C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem  CALL SendTestIndication /?


Beware! - This might lead one to believe its as simple as then providing the param TestIndicationID with a value and the Method would be executed. -- That [ will not ] work!

There is a really [ BIG ] difference between a brand new Instance (copy of a Class definition) which you just Created, and a "Pre-Existing Instance" that has been hanging around since the System started up. The original Instance has been kicked, used, abused and had first access rights to everything in the system. As such it has "Experience" and "History" within the system in which it lives, or exists, and is probably the one you actually want to be using, when you perform a Method Call.

To really drive this point home, some Methods will "refuse" to Execute on brand new Instances, because those brand new Instances are "second rate" copies that probably do not have the Information or Control over the things you want the Methods to operate on.

And you should probably be aware that as soon as wmic.exe is finished with this brand new shiney object, its going to be "thrown away" by the operating system, reaped, garbage collected.. desolved.. and its resources recycled.. hasta la vista.. so your wmic "derived" shiney new object will not be hanging around for very long. In this regard [Static] implies for "this instance in time", very temporary, and "not very useful".. so Executing a Method on a [Static] object is just silly. It's like taking a hammer to a glass sculpture.. its very brittle.

Dynamic objects (or [Non-Static]) objects are much more robust, they hang around and get used alot, they've been around the block a time or two and invoking Methods on these types of Objects is much more useful. Of course they need long term caretakers that "sponsor" them in the operating system ecosystem, so they are not taken out and thrown away by the operating systems recycling system. And that's the job of the WMI Providers that provide WBEMServices, when they startup they create these default Instances of the Classes and keep them going.

To a degree declaring a Method as [Non-Static] saves you time, and helps eliminate confusion, did you remember to refer to the Object you really wanted to execute that Method on? Or did you create a new one and immediately turn around and invoke a Method that is not going to return anything useful?


For example:

This ERROR message can be misleading, because it refers to the CALL verb and not the Method.

This Method is [Non-Static] meaning it cannot be CALL'ed without referring to an 'Existing' Instance.

You 'cannot' call the Method by 'Creating' a new instance at the time of the CALL (that would require a [Static] Method), SendTestIndication is not a [Static] Method. That is what the ERROR is saying, the Class instance provided, that is, the brand new one... just created.. has a Method called SendTestIndication but since it is [Non-Static] it cannot be called on a recently created Instance.

In a way, you made a choice by [not] specifically telling the CALL (which) Instance of the Class you want to invoke this Method on.. by providing no specific way of telling (which) Instance.. wmic "assumed" you wanted to "Create" a whole brand new one in memory, and proceeded to do so for you.. then it <pointed> the CALL at this brand new copy and said { This "one".. use This "Instance" that I just created.. and invoke the Method attached to this "New" copy. }

Its a little confusing, because the Method is attached to the Object just created, but its a matter of "where else" would you logically declare methods that are going to work on this Object type in source code? Group them together.. and we'll make a note the CALL can read later that says [caution! do not allow this Method to be called if the Object is brand new.. aka "static"] If the object hangs around it will be declared "non-static" and CALL will allow that.

The CALL'er is looking at the Parameters proposed to feed the Method and 'saying' { You know what? Nope.. that's wrong, its not right and I'm not going to allow this. Your providing "Invalid method Parameters" and I'm shutting this operation down.. end of story. }

The Class Instance is "new" not "existing" and that is an {'Invalid method Parameter'}

[Instead] a reference must be made to the {one} Instance of HP_WinComputerSystem that was created when the HP WMI Provider service was started on the system. That means a CALL must be made referencing the "already Existing" instance of the HP_WinComputerSystem Class.

In WMIC a reference can be acquired by "prefixing" the "CALL" with a specific "query" such as the following:
C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem where Caption="afs00-2.agnet.tamu.edu" GET /all /format:list
The original "walk" of the CIM properties for this Class "Instance" revealed a property named "Caption" since there is only "one" Instance for this Class possible per machine, the query will return exactly one reference to an "Existing" Instance which will then be used as the starting point for invoking the Method.

If you run the last wmic command and look closely at the Properties of the Object it returns, you will see as "Christmas trees" go.. this Object is much more detailed and decorated with more Values than a shiney brand new copy that hasn't really been around very long. Its probably been hanging around in memory for quite a while and other Classes and services have strolled on by decorating it and updating with useful state information.

It is possible for some Classes to manifest themselves as "more than one" Instance, in which case either the query must narrow the possibilities down to a single instance, or a command line FOR /F loop must be used to wrap the entire call to perform the same Method for each Instance found.

In that case the loop statement effectively "filters" the results and presents only "One" object to the Method invoker at a time, it queues them up and makes everyone stand in line.. so only one Method is invoked by the magic CALL'er wand at a time.

Enabled and disabled, multiple network interfaces on a server come to mind..

When the original "walk" was listed it also provided the current value for the Caption property and it is used as the selector for the instance in the following example. The "name" property is also commonly used.

In reality probably any Instance property could have been used, this one was picked at random.

The results will now return a ReturnValue, but in 'this case' is still not sufficient to achieve a successful Indication event on the HP SIM server.


In this case the Method is invoked, and returns an OUT param called "ReturnValue = 4"

This is actually an Error coming from the Method, and is explained by looking at the Description for the Method
C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem where Caption="afs00-2.agnet.tamu.edu" CALL  SendTestIndication /?:Full

In this case the Method is 'saying' the ReturnValue = 4 means an inappropriate value (not valid) is being submitted by the param 'TestIndicationID=1'

A look at the Description for the MOF of the param 'TestIndicationID' using WBEMTest reveals appropriate values for the TestIndicationID:



Internal identifier of the test event to generate
101 - Informational Test Event
102 - Warning Test Event
103 - Critical Test Event
 And the final result is:
C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem where Caption="afs00-2.agnet.tamu.edu" CALL  SendTestIndication TestIndicationID=101

note: while this will work from a standard console prompt, it will return a syntax error from a powershell prompt. "Unexpected switch at this level." The pre-processing of "quotes" within powershell is especially complex when calling legacy command line tools which also depend upon and pre-process the quotes, which I suspect "provokes" a technically accurate "if inconvenient" syntax error at "one of" the pre-processing times, either by powershell or the command line invoker of the legacy command line tool. I suspect a mixture of pre-processor specific "escape" characters and redundant or re-diracted specific single and double quotes would result in the same result without an actual syntax change to the original statement, simply by accounting for the extra powershell pre-processor oversight.


The TestIndicationID values will correspond to the EventID numbers {1,2,3} available to choose from in the SMH and standalone applications GUI
101 - Informational Test Event
102 - Warning Test Event
103 - Critical Test Event
 

When the Method is called there will be a discernable [ delay ] before returning the [ ReturnValue ] output, the Indications will immediately show up in the HP SIM Management Console as a generic

[ Event Type ] "Test Event"
 
Orange (!)  Major Inverted Triangle for TestIndicationID=103
Yellow (!)  Minor Triangle for TestIndicationID=102
White (I)  Info Circle for TestIndicationID=101


It might belong in a different Blog entry but quickly.

You can do the same thing with WBEMTest.exe

1. open a command prompt

2. type wbemtest.exe


3. press Connect

4. change the Namespace to [ root\hpq ] this is where all HP Providers put their Classes, Instances


5. press Connect


6. from this new Viewpoint, conduct a Search for "Instances" of Class "HP_WinComputerSystem"

7. press the [Enum Instances] button and type in a Search pattern for the Instances of Class

8. there will be a long "pause" while it conducts the search with very little feedback

9. it will return a list of "all" Instances that originated by instantiating a copy of the SuperClass

A. Its important to notice the result box cannot be resized, but all of the line is important when referring to this very specific "Instance", which you will need to "launch, fire, or properly execute" a [Non-Static] Method - so manually copy it down or make a note of it; the cut and paste, clipboard does not work in this tool

B. Close the Query Result and press [ Execute Method..] button

 C. type in the "very specific" Instance line you found before

D. use the drop down to select the [Non-Static] Method of this "specific" Instance of HP_WinComputerSysetem - (method) SendTestIndication

E. press the [Edit In Parameters...] button  (notice: "In" corresponds to the [In] parameters )


F. in the "center" Properties box, scroll down select TestIndicationID and [Edit Property]




G. punch the "Not NULL" Value radio button and type a value of 101 then press [Save Property]





H. then press [Save Object]


I. then press [Execute!]

J. after a very long pause the Result box will appear, and the HP SIM console page will indicate it received an Event from this end point





9/08/2014

HP SIM, sending WBEM test indications


Once a sytem is setup for monitoring within Systems Insight Manager, it will usually have a copy of the Systems Managment Homepage (SMH) service installed.

This is a webpage user interface for accessing data from the HP WBEM providers/agents for intiating various tasks and applications to assist in the mangement and monitoring of the server.

The address for the webpage is usually https://localhost:2381 or https://<name or ip address for host>:2381 (if it is accessible through the hosts firewall).

From the top menu:

Settings >> Test Indication >> Send Test Indication

From here you can select a Type of indication which will use the WBEM subscription between the HP SIM instance and this server to send the equivalent of an SNMP Trap to the HP SIM mangement server.


Including a a lot of useful information in the HP SIM Event which is logged.


Equally there is a GUI tool available from the command line or the Start Menu

C:\Program Files\HPWBEM\Tools\HPWbemTestEvent.exe




The corresponding WMI command line tool is wmic.exe:

And example of how to send a WBEM test indication is:

C:\wmic /namespace:\\root\hpq  Path  HP_WinComputerSystem where Caption="afs00-2.agnet.tamu.edu" CALL  SendTestIndication TestIndicationID=101

The following will log an event to the Windows Operating System Event Log, but it will not generate a WMI Indication to be forwarded as a WBEM Indication. As such it is merely "interesting" and offered for completeness.

C:\> eventcreate /t {information|warning|error} /id {1|2|3} /l system /d "This is an Informational Test Event."

For example:

C:\> eventcreate /t information /id 1 /l system /d "This is an Informational Test Event."



 


iPXE, USB3.0 development platform


Currently I'm most familar with using VMware Player as a host platform for developing iPXE code. To work with new USB3.0 hardware I've discovered I'll need to upgrade the host OS I am using for VMware.

In the beginning I was using Windows 7 as the VMware host, and CentOS 5 inside a VMware virtual machine to compile and a second VMware virtual machine to boot from the iPXE.iso image.

This worked well because the Windows 7 (x64) host could "pass-thru" access to PCI and USB host controllers to one of the VMware virtual machines.

But VMware will not "pass-thru" USB 3.0 host controllers because Windows 7 does not contain a native Microsoft USB 3.0 host controller driver. Further VMware mentions specifically one of the Renasas (NEC) chipsets as "tested" and known to work. Fortunately the adapter card I obtained is a Renasas (NEC) chipset card and should work.

This leads me to conclude that to establish a new USB 3.0 development platform for iPXE I will need to upgrade to a Windows 8 (or a Windows 2012) host operating system and a version of VMware Player that supports "pass-thru" for the USB 3.0 host controller. I am debating whether to use a separate hard disk, or to perform some sort of dual boot scenario.. though my intuition is to maintain a separate hard drive for each purpose. Multiboot scenarios are often hazardous to not only the disk contents, but also the integrity of the boot loader for each and complex disk management while focused on virtual machines can get confusing.

After arranging all of this, it will be very interesting, just to confirm or deny that the original UHCI, OHCI or EHCI emulation (will or will not) work with a true xHCI controller type.

I'm given to understand that (it "might") work, but also that (it "might") not work. I tend to think the latter is more likely.

To work, the new Renasas (NEC) chipset would have to have provisions for the same chipset registers present in the earlier OHCI or EHCI controller types and respond as if it were the older adapter card type in some sort of "fall down" or "fall back" position to support older device drivers.. which would be awesome.. and they have the design rights to make such a device. However it might be in conflict with the later xHCI design goals. And so far I have only tested the UHCI functionality of the USB support added to gPXE before porting it over to iPXE.

I have recently obtained [true] EHCI > UHCI and EHCI > OHCI (Legacy PCI) adapter cards for the purpose of testing the code as close to specification as possible. that is I want to make sure not only UHCI but also the OHCI driver works. But I have not had the time, "the playing board is set" as they say but I have lacked the microseconds necessary in my schedule.

I have also recently obtained a [true] xHCI (PCI Express, or PCIe) adapter card for the prupose of testing the functionality of the "existing" xHCI > OHCI (or lack of functionality). But it requires setting up a different motherboard with a different operating system to support VMware USB 3.0  - "pass thru" for the PCIe card. And then creating a virtual machine with a newer version of CentOS with a newer kernel that also supports the USB 3.0 chipset and bus.  This is certainly doable, its just a matter of time.

Then there are individual USB Ethernet device drivers. So far I have only verified the Asix device driver using the 100 Mbps chipset, there is a second using the 1000 Mbps chipset which it appears a cloner of the ipx/usb-device-drivers Git project is already testing. The results of that test will no doubt be USB 2.0 specific, but most interesting.

As far as I can tell, if the Intel or Via version of the xHCI controller truly deviates enough from the specification that a "different" PCIe host controller driver is necessary, it may be unsupported for a while. I think however Linux may have implemented different host controller drivers to support advanced features unique to variants of the xHCI device (Power Management, TOE, VLAN tagging, Bonding, ect..) and for the more mundane and common features.. it is possible a common generic xHCI driver will support both pseudo xHCI > UHCI as well as xHCI > OHCI variants of the xHCI device.. which would be truly awesome.

To get the speed up through the USB stack however, a rewrite of the USB stack paralleling the rewrite for the Linux host controller device drivers may be in order.

9/07/2014

HP SIM, disabling obscure SNMP Alerts

HP SIM classifies certain SNMP Traps as "Critical" which can get caught up and forwarded as email notifications of questionable value.

Sometimes when searching for their relevance and meaning on HP Forums the meaning is either obscure, or not even remembered by the staff who wrote the software.

In such cases its easier to "find the SNMP Trap" classification within HP SIM and change it to a category that gets "classified" as something that will not be treated as "Critical" and therefore spare the administrative staff of obscure "Critical" alerts.

Here's an example:

Note: "In this case" while the relevance "seems" Critical, a well known bug in the firmware for multiple iLO generations of firmware revealed it was a false indication of the actual state.. so it was not useful, it was in fact wrong. A false indication can effectively "blind" the System Admin and be more harmful than simply turning it off.

SIM > Options > Events >SNMP Trap Settings… >

SNMP Trap Settings

cpqsm2.mib

cpqSm2NicLinkDown

(SNMP) The iLO NIC Link is Down (9015)

The iLO NIC Link is Down. The Remote Insight/Integrated Light-Out firmware has detected the loss of network link.

[Severity: Critical] > "change this to" > [Severity: Informational]