1/10/2017

Programming, windows usb devices

I feel it useful to document by diary/journal when learning something new. In the confusion that exists before consolidating memories are "perspectives" born of a time when I did not know how things actually worked. In that narrrow twilight zone of understanding are common tropes or "pathways" that other people may travel.. perhaps because of a common heritage of prior experience. -- by documenting the misconceptions and "wrong headed" failures.. I hope other people can enjoy the "trip" and arrive at their personal destination (or similar understanding) that much quicker.

USB comes in three versions or "flavors" 1.0, 2.0, and 3.0 generally defined by their speed or data rate, but also defined by the hardware chipsets that enable the ports that physical devices are plugged into a computer.

The wire "signal" is a complex "balanced" overlapping arrangement of virtual bit trains and virtual "frame sets" which indicate chunks of information. Embedded in this stream are timing and data information and multiplexed "purposes" as there are co-existing "different" [types] of virtual pipes with differing behavior. For example, some traffic is "prioritized" over other traffic, such as for pseudo CPU "interrupt" driver processing service. Other types are for "lossy" communication in which dropped packets of data in exchange for "realtime" or "current" data is acceptable (think a telephone voice call where 'crackle or noise' is acceptable as long as the voice is pseudo-realtime and not delayed). Still other types are for absolutely "reliable" traffic even if it is delayed.

Quite a bit of the wire protocol is encapsulated inside the silicon "chips" which are embedded into the computer motherboard or inside the external hubs and usb devices themselves.

These are connected together with USB cabling to provide service.

There are standards for USB cabling and different capabilites with each type of cabling. But I won't get into those feature here.. since I'm more interested in learning to write a program or driver that opens communications and actually makes use of a usb device.

Within Microsoft Windows is an arrangement between user application and kernel applications where by the User Mode and Kernel Mode programs are written using two different sets of Headers and Libraries. Each has their own set of APIs (application programming interfaces).

For the most part these APIs are C/C++ compatible and most User Mode or Kernel Model programs are written in C or C++. Then compiled using either the SDK for User Mode or DDK for Kernel Mode. Each of these "Development Kits" contains their own "complier" customized and setup to reference the correct set of Headers and Libraries at compile time.

Traditionally Visual Studio was used to create User mode programs and Visual Studio was used to possibly edit Kernel mode programs, however when it came time to compile.. Visual Studio would use its complier to compile User mode programs and Kernel mode programs would have to be compiled manually with the DDK compiler.. this has changed back and forth between Visual Studio for both or separately for User mode programs only.. it generally depends on the era and which Visual Studio product is used.

Windows also includes a general purpose "database" which the User Mode or Kernel Mode programs can access at runtime called the "Registry". Historically evolved from a collection of [INI] files into a unified set of binary files to gain speed of access and to allow the database to grow much larger and faster than an ordinary text file based database would make possible. It also allows a certain obscurity or ofuscation (opaqueness) to the database which was later given.. even an access permission control system to prevent certain entries from being accessed or changed without the proper user permissions. All of this means even the User Mode and Kernel Mode programs have parts of their APIs dedicated to accessing the registry, which they access to make use of dynamic runtime libraries and COM/DCOM runtime objects.

All of that is history and window dressing for the main stage of developing application to access USB devices.

In the early days of windows, even in DOS, there were no USB "drivers". Each program had to initate the USB port and manage the abstracted protocol features the chipsets provided to communicated with USB endpoints which represented devices connected by USB cable.

In time however "Kernel mini-port" drivers were created to manage and "share" access first to USB Host controllers, then USB hubs, and finally USB devices. As this "Stack" of drivers came into being Microsoft sought to make driver writing simpler and faster, as well as less error prone.

So the evolution from "mini-port" drivers was to "device driver frameworks" to assist driver writers in creating drivers based on examples or "templates" which guided them in making similar drivers that had similar features.. like debugging or other features.. so that they could all be treated similarly by the overall operating sytem. -- in particular the jump from 9x (VxD) and NT kernel device drivers, was to the WDM - Windows Device Model kernel mode device drivers. (soon to be replaced by the WDF- Windows Driver Framework)... Which led to frameworks for creating all kinds of device drivers.

Specific to USB however was the establishment of the [winusb.sys] device driver which took over much of the job of older or vendor specific device drivers and co-operated with the central operating system kernel to share and manage the USB [bus] as a common "thing" that might be used simultaneously by many programs at the kernel and user mode levels.

USB is a "shared" communications medium, even if it may appear at times to be a cascading set of point-to-point connections. At any one time the total bandwidth of a singular USB port on the computer may be prioritized or divided out among many USB devices on the virtual "bus" shared by all the connected devices.

So once the WDM framework made device driver writing easier and the shared USB bus management made using multiple USB devices simultaneously possible.

Microsoft set about co-operating with the USB Special Interests Group (SIG) or USB Forum, which had shared interests in "standardizing" groups of similar USB devices and coming up with agreed upon minimal protocols to use these groups of USB devices. They were called USB "classes".. not in the sense that they were abstract Programming classes, but that they shared a common set of designs and features which could be probed and expected to exist, and thus similar drivers with "vendor specific" deviances or functions could be accessed.

The win for Microsoft was they could include [in-the-box] with a new version of Windows both a winusb.sys driver for oddball USB devices that were unique, and "Class" device drivers which would provide certain minimal services for "recognized" USB devices.

The USB devices were "recognized" by a process called discovery or "Plug and Play" detection and device driver loading.

USB devices generally have an established protocol where "descriptors" or "strings" of information can be retreived from a USB device after it is assigned an address on the USB bus.

The assigning of the USB address is a base function of the USB bus driver after a USB device is plugged in and powered up.
 Once  the "Strings" are retrieved a search of the strings returns a PCI Vendor ID "like" unique finger print which identifies the device as a possible member of a device class, and a specific vendors product. Depending on the device drivers registered in the Windows Registry or the INF flat file database an appropriate driver or driver "set" of device driver files will be loaded and start execution on behalf of the device in the Windows memory space, in general these are Kernel mode device drivers of the WDM type, a USB bus manager, device driver and class device driver.

The end result is a User mode program can start up and probe the system device list for a target which represents the USB device it needs to peform its function. And by sending IRB (Input/Output Request Blocks) to the Class driver communicate with the USB device.

IRBs are very low-level however and generally a Class driver has a predefined set of headers and library or APIs for communicating with the USB device it manages.. abstracting away (or simplifying) the level of effort required to write an application that uses the USB device.

The really hard part (I think) about all of this is knowing the history of what came before, so that not only you have perspective, but can also "filter" or "ignore" whispers of irrelevant data .. such as information about a VxD, or about a long deprecated protocol like VfW - Video for Windows, which trying to get a foot hold in the current era can distract you.. or prevent you from choosing the most appropriate API to use for the Class driver your interested in probing and using.

There are "models" within "models" describing both "abstract" representations (or "perfect" idealized models of how things work) and then "real" representations of actual devices which have [some] of the chracteristics of the ideal "model" + plus "extentions" or features that the model does not address, which you then have to figure out how to address. Some Class drivers build in (vendor specific) methods to access these other features.. but some do not.. in which case you may have to consider writing a mini-port driver to co-operate with a Class driver (an extension "plugin") .

The USB forum (or SIG) also describes many of these same things in similar but not identical terms, so that an abstract model in Microsoft documentation may not line up with the description  of the same thing in a generic USB forum Spec written with the idea of serving multiple purposes beyond Windows programming.. such as programming for Linux or Apple OS products.

The USB Specs are also [very] hard to read as the knowledge base of the authors is both jargon filled, presumptive and lacking in details, and written "backwards" or "upside down" in that they are written more like legal contracts preceeding object of discussion with adjectives or descriptions "before" refering to what they are actually talking about. In Germaic descendent languages like English, it can be very hard to [overlook] or [ignore] the fact that the document appears to have no purpose or subject matter, until "after" the discourse on the descriptions and actions.. leaving motivation to "finish" reading the Spec document..very difficult to find. (I believe) certain speakers in other languages will have an easier time reading the documents as I have seen similar "speech" patterns in other languages.  -- once "consumed" however Germaic readers can generally "re-organize" the material almost subconsciously within their mind.. much as when translating a foreign language internally.. and its easy to "forget" how hard the first reading of the material was to understand.. a certain amount of "faith" or experience in readin RFC or Spec documents may make this easier.. but for the inexperienced or "uninitated" newbie.. this can be a very difficult barrier to overcome.

In the perfect Microsoft Windows programming world. A USB device will have a high level, high functioning Class driver and the end user User Mode Application will have an available "framework" specific to their task.. such as DirectX Direct Show for Video, and the end user will not have to write a device driver, or understand many of the details of how the actual silicon or the device performs its task.. it just appears to do so..  like "magic"

A further "funny" detail is that you should not loose sight of the fact that a USB bus is a negotiation between two independent devices arbiting over communcations and services. The USB device is itself its own computer, with its own operating system (however simple that may be) and thus has its own programming. Often called "firmware", it can have bugs, it can change, and features that were available in one version of the firmware can change or be removed in the next.

Generally the USB descriptors are used as the IUnknown or COM type object feature of "discovery" and "enumeration" or list of available features.

It is through the USB descriptors a USB device provides about itself that Class drivers can setup and expose features of the device to the operating system user mode programs.

In the inital release of a hardware USB device, the firmware may not be 100 percent complete. And the USB descriptors may describe features or functions that are not quite bug free, or even implemented yet.