Raspberry Pi Zero W - Streaming Webcam

I've been swamped by travel and things to occupy my time. But the $10 dollar Raspberry Pi Zero W has been on my list of things to do. I wanted to reuse a Logitech Webcam and use the Pi Zero W as an on demand streaming server.

The Pi Zero W is basically the circuit board with the empty row of header pins in the black plastic box with a clear top and sliding door to expose the header pins, if installed. It has one microUSB power port, one microUSB OTG power and communications port. And a microHDMI port to the right and a microSD card port on the right end. On the left end is a Pi Camera port.. which I didn't use.

I used a microUSB OTG to Type A USB 2.0 adapter, which immediately made a Left turn and blocked the lesser used (power only) microUSB port. The adapter has a pass thru microUSB port which is used to power the Pi Zero W.

The MACBOOK multi-function Lan Adapter is hanging off the USB 2.0 port and provides an Ethernet port, two Type A USB 2.0 ports, a full size SD card slot and microSD slot.. as well as having its own external power port.. should other devices need more power than the Pi can share over USB.

Both the multi-function adapter and the webcam get their power from the Pi over USB.

I have tested this with POE to microUSB and it works fine.. so CAT5 cabling to remote locations is possible.. even where WiFi is not available.. and there is no high voltage mains available.

The Noob or Raspbian Debian distro based on Jessie is the default start up Operating System and provides an LXDE desktop and a lot of apps in a meager 8 GB. I use a 32 GB microSD card and win32imager to write the raw disk image to the microSD card. The reason for over provisioning is write level wearing means the card lasts longer the more extra unused space is available.

Networking is configured to start up via DHCPv5 but has a flaw.. if the DHCP server isn't available a hard coded -1 option in the daemon startup defaults means it quits and doesn't continue to try to grab a dhcp address.. so very often in edge conditions.. the Pi will fail to obtain an IP address... and never try again.

The solution to that is hard code an IP address in the /etc/dhcpcd.conf file at the bottom of the file as specified in the dhcpcd.conf man pages. Doing the same for wlan0 might also be advised, as a backup measure.. but be careful of routing loops or duplicate default routes.

The Ethernet  setup involves

1. editing the /etc/dhcpcd.conf

pi$sudo vi /etc/dhcpcd.conf

.. and placing something like the following at the bottom

interface wlan0

static ip_address=

interface eth0

static ip_address=
static routers=
static domain_name_servers=

The WPA2/PSK setup involves

1. scanning the WiFi air space for SSIDs

pi$ sudo iwlist wlan0 scan | grep ESSID

2. using the wpa_passphrase command to generate a configuration stanza

pi$ sudo wpa_passphrase "testing"

.. which will prompt for the WPA2 pre-shared key (the WiFi password)

3. copying the output lines from the wpa_passphrase and putting them in

pi$ sudo vi /etc/wpa_supplicant/wpa_supplicant.conf

.. after all of the lines already in that file

4. then performing a reboot or manually command ifup wlan0

pi$ sudo ifdown wlan0
pi$ sudo ifup wlan0

Setting up the Pi should always begin by running the command

pi$ sudo raspi-config

Its a whiptail (menu of menus) traversed with the keyboard and no mouse.

A gui version exists, but is incomplete.

Raspberry amd RealVNC have a shared history and Pi with Jessie now comes with a free Home use Cloud edition of RealVNC which can make the Pi desktop a "server" accessible from anywhere in the world.. similar to the Teamviewer style shared desktop programs over the Internet. One simply downloads the RealVNC "viewer" and it starts a cloud "address book" of all the RealVNC servers you have configured to attach to your cloud addressbook when they come online. From this addressbook a live connection can be established in spite of firewalls or nat routing challenges.

Every menu option should be looked at, the Pi is by default tailored for Great Britain and that locale, keyboard and timezones.. and many other options are not what you want to be using anywhere else in the world.. even the WiFi radio bands are configured here.

If you decide to disable the X-Windows server and run the Pi by command line only, the option is in the menu. If you want to start the SSH server, the option is in the menu.. all the basics you'll want to change are in these menus.. explore and be aware.. its cleaner than hacking at the infrastructure based on experience with other distros. (BTW) sshd is disabled by default, and permitrootlogin is set to prevent it.. to change either you'll need to visit the /etc/ssh/sshd_config file.

All of this to say you'll probably need a microHDMI and  and usb mouse and keyboard. I got along using a logitech RF mouse and USB nubbin to connect that to the Pi, then a Bluetooth keyboard using the Bluetooth systray icon to pair the keyboard with the Pi Zero W.

There are "unsafe" ways of post-writing the Jessie Pixel image to the sdcard (to) mount the filesystem and configure a few things.. but the hazards are real and time consuming.. go the monitor and mouse, keyboard route and it will go much faster and safer. Use the raspi-config command !

After all that you might want to apt-get update, and possible apt-get upgrade your version of the operating system; and use apt-cache search to find useful packages.

Not all useful packages are in the Debian Jessie repository however, some may need to be added to get packages of a more unique nature.

And some code drifts in and out of development favor and is only available by subversion or git repositories and must be compiled.

apt-get install vlc

is a nice one to have

mjg-streamer is also very nice to setup a streaming webcam, without fswebcam, ffmpeg or vlc but parts may support it.

This Raspberry Pi Forum article details how I got it working, following their directions and finally fixing the startup script main command line to avoid a typo and add the path to the webserver content.

mjpg-streamer: Blank Image and No Video

The result was a working streaming on demand server with "near" realtime framerate (24 fps), that starts automatically each time the Pi Zero W reboots.

The pages also demo the different client types, static snapshots, browser streaming, java applet streaming. javascript streaming, VideoLAN client streaming and finally a demo control panel for changing the camera UVC controls on the fly while streaming.


Czur, the SUNPLUS Burn file format

The ET16 scanner is based on the Sunplus or iCatchtek spca6330 image processing board. Many vendors use the same hardware components. The EKEN ("eee-ken") action camera also uses the spca6330 board and a Windows PC hosted firmware flasher called the FRM ("firmware resource manager") to perform ISP ("in system programming"). The FRM can also produce Card ISP files ("insertable memory card in system programming files") . The Sunplus BRN (""sunplus burn files") are memory "Card ISP" files

These can be used to update firmware in production devices capable of being placed in RSV mode ("read save verify" mode).

Embedded hardware comes with NAND, eMMC or SPI accessible EEPROM memory. This is where the firmware is stored. Typically its divided up into chunks or "partitions" which are accessed by the bootloader at the start of memory when start up commences.

If a generic operating system like Linux is used the path of least resistance is to format some of the partitions as "virtual disk drives" with a well known file system such as ext2, or a compressed file system like cram or squashfs. In that way a generic monolithic program like Busybox can act as many parts of the operating system in a single binary to save space while exposing a common and familar (and scriptable) toolkit and programming interface.

[- then again on second thought nothing precludes "burning" as part of a partition a small file which contains a partition "image" of a filesystem itelf, like a fat16.img file.. -]

If a more conservative and deterministic (aka reliable) or 'time critical' operating system is used like eCOS, RTOS or ThreadX then the memory chunks or 'partitions' may simply be "mapped" regions of storage accessed by the programs themselves without any file system.

The SUNPLUS BURN FILE format appears to be more of the latter type.

There appear to be three partitions, two general "resource" partitions in which general purpose tool binaries are placed, which can be anything from graphical image files to font files or actual programs.

And a third partition dedicated to "default firmware", which some might call a "bootstrap" or "boot loader" except that in this context its actually more like a combination bootloader and kernel.

Resources from the other two partitions are accessed by the "kernel" using a known offset in persistent memory devices always present on these "known" hardware component boards.

The exact architecture of the file formats is not known, however they are probably a type of ELF file format, and given the processor is likely an ARM A9 it is probably for a 32 Bit ARM processor architecture.

Its more common these days with multi-core or multi-processor chips to run two entirely "different" operating systems on each core.. one dedicated to a "time critical" -- real-time -- operating system like ThreadX, and the other to a general purpose (or easier for 'humans') to script or program [user-interface] operating system like "linux".. this is called [a]symmetric multi-processing since they do not neccessarily run in synchrony and rendevouz ("link-up") by passing messages back and forth through established methods in their operating systems.. its more like the worlds "smallest" internet of computers. - This does not appear to be the case with the ET16 scanner.. if it is running two copies of an operating system, they may be the same operating system. (I base this on observing both partitions appear to simply contain binary blobs and not a recognizable 'linux' or 'dos-like' file system).

While it is possible to use the FRM tool on a production device in RSV mode to "download" or "retrieve" the actual burned firmware and make a "backup" before flashing new firmware, or for further development and inspection.. I have not done this with the ET16 scanner.

When the binary "blobs" are retrieved in this fashion they come back and are saved on typical Windows file folders. And can be further examined.

My purpose is not so much to "reverse engineer" as to simply understand in "general" terms how the ET16 actually works.. and so much of this is from inference and observation than actual "hacking".

Besides, the Sunplus Burn File (SPHOST.BRN) or "System Programming for Host Burning" file is already provided as a "firmware update" for the ET16 as a download from a website.

"Unpacking" a [Sunplus Burn File] should be a matter of looking at each "partition" inside the singular Burn file (unfortunately not a file system you could mount) and extracting the individual binary ELF format programs. - but then we'd have to stop, since unless you had a sufficient debugger or hardware development board. The programs would not run on a PC.. perhaps inside a virtual machine like QEMU, or debugger like IDA Pro.. but that would pretty much be end of story.

One gentle man has used Python to create a few scripts to extract the ELF binaries from a Sunplus Burn File

Reversing, Exploiting und Patching an Action Camera

Martin's GitHub for SPCA-FUN python scripts

One thing that came to mind was as a 32 ELF the byte-ordering might be reversed, and this might be the case in the final Burn file as well.

I hope this satisfies a few peoples curiosity.

p.s.. The EKEN (pronounced "eee-ken") action camera uses the "resource" partitions to store "parameters" or settings which the other programs access and use while running. By tinkering with these settings they have enhanced or modifed features and behaviors of their action cameras. This is a dangerous thing to do. And it does lead to "bricking" or making their $300 to $400 cameras useless.. but explains much of the interest in "hacking" their devices.

p.p.s. Sunplus is a collection of companies in China and has a large and deep history with good documentation and a thorough patent portfolio. Not all of their documents have an English language counter part. And many of their documents are proprietary and confidential. The few public samples I have seen from many years ago are excellent and quite detailed. (I am under no NDA). Sunplus does responsible releases of public GNU code that they use, but do not include the specific binaries and scripts they are not beholding to release. -- That said reverse engineering to bring much of the hardware based on their technology under support in Linux is always ongoing and appears to take place mostly in Europe. -- From I can see Czurtek is probably under NDA and cannot release many specifics about the actual EVK software development code and hardware interfaces. They have however built a rich API on top of the proprietary code and expose what appears to be a very useful API of their own.


HP Folio13, How to Fix the WiFi button

Hewlett Packard made a laptop called the HP Folio13 a few years ago. The WiFi would work for a while and then appear to stop working (even if the WiFi keyboard key was On). This is how to fix that. Basically "only" the HP Connection Manager could turn a Secondary "Airplane Mode" switch off and on inside the laptop. Later releases of the drivers on the Laptop Support page did not include the older HP Connection Manager tool. -- If Airplane mode was turned on, and the drivers upgraded, you no longer had a way to turn the Airplane mode off, or re-enable the WiFi internal switch.

Demo of Flipping the internal WiFi Airplane Mode Switch

This internal switch is [independent] of the keyboard WiFi switch, which will remain On at all times whether the Airplane mode is turned on or off. It is also "counter" intuitive, but Airplane mode ON, means the internal WiFi Airplane mode switch is turned OFF. And the two are not "Exclusive".

You can turn the Airplane mode ON, and then manually select [only] the internal WiFi Airplane mode switch independent of the internal Bluetooth Airplane mode switch. (AND) the internal WiFi Airplane mode switch "is NOT" labeled as a separate switch.. but it behaves as a separate switch.

The proof of this, is that the ERROR condition when using the Microsoft Windows Troubleshooting tool indicates the WiFi Radio is OFF and instructs you to turn it on, to resolve the problem. Which is NOT possible, because it is already ON.. toggling it back and forth ( regardless of the lighted LED state ) has no effect.

The keyboard WiFi switch has a white LED in the upper right corner of the keycap which remains ON at all times while the internal WiFi Airplane mode switch is "flipped" back and forth using the HP Connection Manager tool.

When the keyboard button is actually Turned OFF the HP Connection Manager displays a [Totally] different "kind" of OFF. ( this is "Real" Off, not "Airplane" Off )

Airplane mode OFF is much more subtle and "mis-leading" when the WiFi button is ON but the internal WiFi Airplane mode is OFF - it merely says "Off" and not "Disabled by Wireless button"

Technically, from my investigations it is a (problem with the Bluetooth sharing the WiFi button) in the programming logic for Airplane mode. You can switch WiFi or Bluetooth Off or On "while" Airplane mode is Off or On. This drums up the possible combinations to at least 6 and possibly 9 "modes" which the display interface does not accomodate. And while they "tried" to cover all combinations.. the user interface failed to do so.. and the Intel Connection Manager doesn't even try.. it just fails. -- dumbing it down.. just make sure to turn the internal WiFi Airplane mode switch OFF or On from inside the HP Connection Manager.. there is no other place you can control it.

[AND] don't try to control the Bluetooth state from the HP Connection Manager, only use the keyboard button. -- This is fundamentally what broke everything.

Upgrading the driver from the HP Support page for this laptop installs a "new" Intel Connection Manager, which cannot disable or manipulate the internal WiFi Airplane mode switch on this laptop.

You can still uninstall the Intel Connection manager, then use Windows File Explorer to browse to the C:\swsetup directory and find the specific HP spXXXX package directory and a subdirectory under the the new Intel drivers.. (avoid) installing from the top level directory that would also install the Intel Connection Manager, instead go to the drivers only directory and look for [ dpinst64 ] and double click that .. it will [ONLY] install the updated Intel WiFi drivers with NO Intel Connection Manager. Then find a copy of the older HP Connection Manager installer and install that to [regain] control over the internal WiFi Airplane mode switch.

Beware!! the fully opened HP Connection Manager cannot be used to turn the internal WiFi Airplane mode switch off and on. [Only] the right-click system tray icon (round blue ball) and going straight to the WiFi switch under the Airplane mode menu actually works.


Kdenlive, a replacement for FinalCut Pro or Adobe Premiere

Opensource on Windows usually doesn't blend well. But Kdenlive is a suprisingly functional and smooth NLE - Non-Linear Video Editor that just got ported to Windows and MacOS. Free as in Beer it has all the features you would expect in the old FinalCut Pro, Adobe Premiere or Sony Vegas Movie Studio products. Multi-track, themeable and stylable its also easy to adapt the editor look to something more comfortable. kdenlive.org

It has a ton of transitions, color correction, and video and audio effects. Its built upon several ultra-portable opensource project foundations which doesn't tie it to one PC platform. Originating on KDE on Linux, it runs just as well on many vintages of Microsoft Windows and Apple MacOSX.

If your in the market for a video clip editing tool of the calibre of FinalCut Pro or Premiere, definitely give it a shot. On Windows its still so new there isn't even an installer. Its just a smallish Zip file to download and extract, and a few ffmpeg files to copy into its core program folder, and a presets folder to copy from the ffmpeg distro into its core program folder and then simply double click the kdenlive.exe - there is a warning box that currently flys up and suggests some missing icons and add-on tools.. but they are totally unnecessary. The default windows MFC control icons are both familar and makes it feel much more "native" to the Windows operating system.. and I presume the same would be on MacOS.

The Linuxscoop YouTube Channel reportedly uses it to edit some of its videos (if so):  then take a look at that channel to see if its what you've been looking for -

I would caution (a) it's "Zippy" as in fast, and (b) its not a "Storyboard" editor like Windows Movie Maker or iMovie.. those are [cut and paste] editors that let you draw a story like a train with box cars.

A Non-Linear Editor, gets its name (NLE) from the ancient artform of literally cutting film into strips called "clips" and then resequencing and splicing them together (either) one after another or even (on top) of each other to "composite" an entirely new visual scene.. this was the way Special-FX were done in Hollywood back before there were "computers" and even how the first Star Wars movie was made.

As an NLE with infinite [tracks] both above and below, Kdenlive.. can composite.

[Note: I got a an out of bandwidth question about setup. It does (Not) have a slick installer yet, its still just a Zip file download. After downloading on Windows, extract it. And download the ffmpeg Zip file which is linked from the same page. Extract ffmpeg, inside that folder are ffmpeg files, copy those over straight into the Kdenlive folder, [and] copy the ffmpeg "Presets" folder over into the Kdenlive folder. Then you can double-click on the > Kdenlive.exe and everything starts up and runs just fine. -- understand Windows isn't their primary focus.. it just happens to run extra sweet on Windows.. enjoy the Candy.] - {You add Tracks from [File] > [New] > mid-page > "Video tracks" or "Audio tracks"}


Czur, How to Scan Books

Scanning books begins with capturing images using an image scanner. It also requires lighting and an environment that presents the pages one after another. When complete the operator is left with a stack of images that require processing.

During processing the images may also be read with character recognition software to create  keyword content that represents the image content. This means images can be inspected and recognized as containing language text which can be used to replace the image content in a new image-less document format, or used as a keyword map to find coordinates in the image of the page.

Finally images are bound into an electronic doc, like an ebook.

These are the steps to transforming a real book into an ebook.

A couple things to consider:

Choosing an image scanner and a lighting environment and scanning hundreds of pages per session can be a time consuming and difficult task.

Choices for the scanner are many but may be limited by available budget.

The Czur ET16 scanner is one scanner that costs less than many images scanners. It includes a lighting system, hand/foot controls to remote trigger an image scan (and autoscan which can detect when a page has been turned and automatically trigger a new image scan) and includes software to perform post processing, optical character recognition and binding into portable document format (PDF) ebooks.

It arrives as a kit that requires some assembly. The software is downloaded from the internet and installed to a Windows PC. During installation the software is activated using an activation key attached to the bottom of the scanner. The Windows PC is connected to the scanner by a USB cable and turned on. The Windows PC recognizes the scanner and completes device setup.

Scanning a book can take considerable time. It may require more than one session. Each session has an associated date and time when it began. The software tracks the date and time when each session begins by creating a new folder on the file system for each session labeled with the date and time the session began. 

To scan a book one begins by setting aside a block of time in which they are less likely to be interrupted. The Windows PC and scanner are moved to a darkened room with lighting less likely to interfere with the light provided by the scanner for capturing images. The background mat is placed below the imaging sensor and lined up with the edges of the scanner.

The Windows PC is relieved of extra running processes and the Czur software is started. A USB cable is connected between the Windows PC and the scanner. A choice is made to use the hand or foot switch  ( or to rely on autoscan ) and if necessary is attached to the scanner. The scanner is switched on.

After choosing to Scan rather than to Present.

The Windows PC running the Czur software will display the 'work table' for performing post processing, optical character recognition and binding. Before these tasks can be initiated a Scanning session must be performed.

The upper left menu [Scan] button is selected and a 'sub application' is started that searches for a scanner connected by USB cable to the Windows PC. A live 'field of view' is presented in the center of the screen which shows what the scanner can see. 

The right shows the current directory listing of the current 'date and time' session folder. 

The book to be scanned is placed in the scanners 'field of view'.

Scanning a book cover will not require holding it in place. Scanning the interior pages of a book may require holding the book open using fingers or hands.  The scanner will automatically process an image with color drop-key to remove fingers or hands wearing colored drop-key covers while holding a book in place. Color drop-key removal will only be performed for fingers or hands found to the left, right or below the page content.

When a scanning session is complete the [X] in the Upper Right is clicked to close the 'sub application' this will bring all of the scanned images to the work table and populate the Right navigation panel with their names.

As each image was scanned it was copied to the PC and given a filename. As each filename arrives on the PC it is "processed" based on the "profile" that was selected for the scanning session. A 'pro-file' defines how to 'process-files' for a session.

A work table full of scan images can immediately be bound into a PDF or TIFF file by going to the Lower-Right and pressing [all] to 'Select all' images. Then going to the Upper-Left and pressing 'Archive' and selecting [as PDF] or [as TIFF]. A request for a destination and filename will appear and create the 'Archive' document.

PDF and TIFF files are 'multi-page' documents which can be viewed in a PDF viewer like the Chrome browser, or an Fax viewing program.

While the above description is the simplest workflow.

There are many variable settings, controls and tools which can be used at each step to attain greater control over the overall process and final output.

Global settings such as Scanned Image resolution, Image file format, dpi and Left/Right page splitting order are chosen from the Upper Right [Settings - Gear] icon.

Session settings such as Color Mode (a combination of Brightness, Contrast and Sharpness) are selected from within the Scan 'sub application' to the Upper-Right just before scanning. And they can be tweaked somewhat by disabling [auto] exposure and manually moving the slider control underneath the preview image.

Session settings such as picking a Profile (a combination of deskew, autocrop, dewarp, page-split or none) are selected from within the Scan 'sub application'  to the Lower-Right just before scanning. Tweaking is performed upon the image files on the work table after 'sub application' is closed.

When working on images on the work table, a single image can be tweaked at a time with the controls underneath the selected file image. Multiple images can be tweaked in batch sequence by selecting all the images making up a batch group and then prototyping the changes to be applied to all of the images in the batch group from the [Bulk] menu option.

Archiving allows creating PDF or TIFF files, or OCR doc file and optionally combining those with PDF files to create a variant of the standard PDF format. These PDF variant files can include a keyword index that maps to locations within the image pages in the PDF file. This makes the PDF image document keyword 'searchable'.

The end result of a scanning session is always a multi-page Archive PDF or TIFF. The scanned images from a session are removed to ensure space is available for the next session.


Czur, skinning with DuiLib

The Czur windows software has an easily modified user interface. I noticed the choice to use DuiLib when I went looking to see if I could change the keywords with Astrogrep. That revealed a simple xml text file with Chinese to English mappings. Then I got curious.

The "skin" directory contained a series of xml files and png files which looked very regular. Also the binaries included a dll library called duilib_ud.dll googling did not turn up much, but a very few hits from 2005 pointed towards a Chinese opensource project to re-use code from an effort to make user interface design simpler and better. Microsoft had released wpf - windows presentation foundation, but most felt it was too heavy and difficult to learn and that most people would not use it. Internally Microsoft was known to use simpler easier to use tools called "Window-less Controls - also" they did not release to the public. So one person Bjarke Viksoe  released an opensource project that emulated that "known but unobtainable" framework. The Chinese project DuiLib Group took this and built upon it. It was called DuiLib - Direct User Interface Library. It appears to be cross platform, adaptable to MacOSX, Linux and Windows.

DuiLib takes an XML file and a bunch of graphic files and creates controls and pastes them on a single Window which shares its window to the Window-less controls which users can interact with. Sometimes its promoted as a "window-less" user interface in that it doesn't exactly emulate the modal windows that a Microsoft Windows program might display. But instead produces "canvas" with controls layered on top. This makes it simpler in that its more like a webpage, and many of the hidden behaviors of more complete user interfaces are not available to confuse people who would not typically use them. So.. its rather like making "Pizza".. and everyone likes Pizza .. right?

So the idea of the "Skin" is just like a layered "Pizza Topping".

Hacking XML files in a text editor is not a lot of fun. Fortunately the team (called the DuiLib Group) also created a GitHub account for DuiLib Modify and released a toolkit with sample user interfaces, the duilib source code and a graphical GUI designer with a Visual Studio 2008 .sln (solution file).

That meant I could install Visual Studio 2008 and compile both the duilib library and the DuiLib Designer tool, and use that to Graphically study the Czur XML skin files.

The only problem that came up was that DuiLib is mostly documented in Chinese. And I speak English. A lot of the source code was in English, but the user interface controls were labeled in Chinese. I used a hand held visual translator to look at the labels on the screen and translated those into English by hand and compiled the DuiLib Designer as a binary release. Then I created an Application distribution project added to the VS2008 solution for DuiLib and created an install package.. Setup1.exe... and put that on the laptop attached to my Czur scanner and installed it.

It worked fairly well, though I do not have a project file with the Czur skin, DuiLib Designer can read the XML files and import all of the graphics and render the canvas panels on screen.

It is not perfect and I still have a lot to learn about the Designer. There are docs, in Chinese for it.. but I don't really plan to go as far as to create a Project. Mostly I just want to relabel controls and perhaps change the colors of a few elements.

I am not sure this could be done on an Apple Mac, and that software while close is not finished. But it might, there is no reason to think they did not follow the same design principles.

And it kind of points out this may be the direction they follow in bringing Czur to the Linux platform.

DuiLib is opensource, and its platform friendly.. not "exclusionary" or "clique-like". It has broad support and appeal for billions of people in China, and now maybe for the rest of the world. Learning about the Czur skins seems like time well spent.


Czur, bookscanner Language translations

Czur scanner keeps delivering surprises. I found the user interface is based on a framework with a skinnable template. Inside the skin folder was a single Chinese to English mapping file. So it was easy to change the odd or difficult to understand word choices into something better. - since then I've recomposed most of the Scan dialog window and Publish and Bulk edit user interface windows using the translate file as a personal notebook to record my discoveries. Its gone pretty good.

Soon I hope to condense what I've learned into a few short video tutorials in English for anyone interested.

My thoughts are both jumbled and excited, but they are also getting simpler.. distilling what exactly is or will be possible and what will not.

The scan dialog reduced down to just Color Modes and Profiles. The scan resolutions are preset in the application itself, but also limited by the defined capture modes pre-programed into the optical sensor device and its onboard embedded computer. In general there is a location to make a single choice for the default resolution, which is pretty good. Custom choices are possible, but the default is dense enough that other than using it for a microscope it should be sufficient. Lighting, Gain, Contrast and Brightness adjustments, also while possible are preset in the Color Modes. So when capturing they really are not a concern.

Once a single image is captured with a scan, it is uploaded across USB to the computer, where the chosen "Profile" for the session, continues post-process the image files into their "temporary presentation mode". But they carry all of the original captured image information in this form so they can be reverted back to original form and re-converted into other Color Modes or Profile types. What actually occurs in the "post processing" depends on which Profile was selected.

For example the one for single flat pages does far less, than the profile for double bound pages like those in a book. Where the single flat profile will straighten and crop and stop. The double profile will square the curve linear lines projected by the laser beams, then construct a map of the dual page surfaces and attempt to flatten the pages. Then that profile will automatically cut or "split" the single page into two separate files.. and depending on a global option as to which should come first in a stack.. Left Page then Right Page, or vice versa.. each is written out with an automatically generated name. Right Page then Left Page order is also a selectable global option.

After capturing scan images into these "temporary" session files, closing the Scan window takes one back to the worktable, in which the temporary files appear in the file list panel to the Right.

They must be individually or in 'Bulk' selected in order to perform a "Publish" operation.

The Publish menu choice then offers to (a) create a simple PDF (b) create an OCRd and indexed, therefore 'searchable' PDF (c) create only an OCRd searchable index or (d) create a lossless multipage TIFF file with or without compression.

Lossless is better for archival purposes. The original captured image is used to create the second generation lossless copy, but the degradation stops there.

If you attempt to close the worktable before Publishing, it will warn you that all of the session scan images will be wiped from your work space and will be subsequently unrecoverable.

On the worktable are tools to basically "tune" or "convert" choices made when the original reference scan images were captured. These can only be used on the current session scan images. While other previous session images, or other suitable images can be included and inserted into a current session document, they cannot be edited with the tools in this session.

A final menu option is for performing each "tuning" tools action on a group of selected scan images sequentially one after the other in a Batch mode. This is computationally intense and time consuming, but the batch function provides a scheduled job list and task schedule to monitor the situation.

Other document or ebook creation software packages should be fully capable of working with the session scan images and exported or saved TIFF files and go further in a more comprehensive archival documentation project.. for example if adding EXIF meta data, or sRGB, adobeRGB color space information is important for you... you can with supplemental software if your willing to purchase and learn how to use it. The Czur kit however will serve as ample low cost training wheels if a $500 or annual subscription to Adobe products is not your first inclination.

It is important to acknowledge that due to practical USB transfer speed, and most peoples patience, the scanner only provides access to JPEG compressed images of a relatively high resolution. Saving them as TIFF will insure they do not loose any more to generation loss.

The scanner is also a UVC compliant camera, and exports USB Endpoints, some which advertise YUY2 ... so 'lossless' images of lower resolutions are possible.. if that is a real need.

I have had a chance to preview the OSX version of the scanner software and it appeared to support a similar work flow.. which is very generic, well supported and well understood.

I have also had a chance to preview the Twain32 driver and it works well.

And I have seen, though not worked with a set of exported functions which would allow any program capable of Win32 COM calls to take control of the scanner over USB and perform most of the same operations as the provided application software.

Detractors are that there is a language barrier, and it is not a finished product or project. Firmware and application software is still being developed and released and its support base for other operating systems is still expanding. It is a USB video class device so there should be no limits.

But on the other hand it is extremely "open" to end users and Third party contribution and development, other than tiny and specific technical details and specifications of interest primarily only to the hardware manufacturer it is very accessible. An SDK, Windows App and Windows Twain32 driver, Apple MacOS software.. and a generic UVC interface make it a very attractive learning project.

p.s. I took a look at the MacOS El Capitan [Beta] software as well and it seems to be a perfect feature match, minus only the Save as TIFF feature for the moment. It is not bug free, there are a few bugs, but with patience and experience it is very usable. The "skinnable" user interface Language translation files are also there "in a way", they are not XML files, but plain resource files. I haven't played with them yet.. but if they behave the same I could likewise adapt them for myself.

There are a few reasons for pursuing a Document scanner/Book scanner at the low end of the market. First they have dramatically fallen in price and the software and methods have been time tested and collapsed into a very few steps. Where we used to worry about "eternity" we now generally only worry about the next few years and accept the archival formats will not be "perfect". So archiving a cookbook, school book, or favorite magazine seems much more likely. Dressed up as a stylish desklamp or as more functional furniture than high speed book "ripper" it resonates practicality.

Some might ask why not use a "Point and Shoot" camera, or a moble phone.. but the problem there is one of lighting and consistent repeatability. A camera or mobile phone lighting stand could be tossed into a closet and only pulled out when needed. But since this already looks like a "smart" desklamp.. why not use it as a document scanner as well.

For the cubicle bound office worker this is especially appealing since options for local storage come down to a desk drawer or precious overhead storage for the elite. In which case the Czur scanner as a personal desktop option always ready for use is especially appealing.


LVM, Virtual Volume Management

I just got back from renewing my training experience with Red Hat Linux.

Its always full of new stuff, and this time it included details about RHEL 7.0

We haven't fully adopted RHEL 7.0 yet, but its on the horizon. If RHEL5.0 were Windows XP then RHEL6.0 would be Windows 7 and RHEL7.0 would be Windows 8 or 10. There is so much change.

But one of the things that hasn't changed is the use of LVM.

LVM stands for Logical Volume Mangement, and I had a revelation.. which I have not had before. I know not everyone 'gets it' and some even fear it. But after this 'new point of view' you might like it too.

I was sitting there in class and noticed that LVM parallels what we do with fdisk/gdisk, make file system and so forth. It was like the older partitioning tools were handling the 'Physical Volume Management' or 'PVM' where the 'LVM' tools were handling dicing up the Virtual Volumes that were created from the 'Physical Volumes'.

Its not really quite as simple as that.. but it forced a tunnel vision like focus.. that really LVM was just a 'terribly' poorly named system of tools for performing this 'Virtual Volume Management' function.

LVM truth be told is not its full name, its actually LVM2 because there was an earlier attempt called LVM1 or generically 'EVM' for 'Enterprise Volume Management'.

Enterprise Volume Management was needed because the 'Enterprise' could not afford down time, and needed to speed up the maintenance of replacing drives, extending or shrinking file systems on a live system.

This is critically important in Server class hardware systems, but why is it also useful to Desktop users?

Because the same abstraction allows re-swizeling or reallocating more or less drive space to 'Volumes' which can contain either the entire '/' root file system or any compartmentalized portion of it, like a /home branch.

Its also really nice when drive sizes out strip the ability of BIOS or a particular version of UEFI to access a block on a physical disk and need a shim, or driver provided by the hardware vendor to make it accessible.

The crux of the 'learning curve' however for newbies seems a pathological 'need' by instructors to make it sexy or 'include more stuff'.. usually tacking on things like MDRAID or synthetic software RAID, or more resilient file systems like journaled file systems in the discussion.. distracting and confusing.. and blending the information into this monolithic mess.. that leave a lot of smart people thinking they are inherent or part and parcel of the LVM system.. they are not.. they may depend on LVM to some degree.. but only as much as a file depends on a file system (any file system) for storage blocks.

So what is LVM ?

Put simply, its fdisk for Virtual Volumes.. or fdisk for Volume Groups.

You see to create a Virtual Volume (aka a Volume Group) you first need building blocks, these are called 'Physical Extents' or 'PEs' in Physical disk terms they are 'blocks' of storage. They can be made from carving up whole disks, or carving up MBR partitions of disks, or GPT partitions.. it doesn't matter.. its an abstraction of the 'Physical Blocks on Disk' to virtual 'Physical Extents'.

Once a whole disk or partition has been carved up into PEs they are then 'used' to compose or build a 'Volume Group'. (and you can 'size' these PEs independent of the block sizes on the Physical disks underneath this abstraction, at pgcreate time you can define the PEs 'size' in bytes)

A Volume Group then is like this 'Virtual Hard Disk' which like a thanksgiving turkey, needs to be carved up into smaller Virtual Partitions before you can use them, or mount them in your operating system. Those Virtual Partitions are then called 'Not Logical Partitions' but 'Logical Volumes'.. I can hear you scream.. why.. but why? are they not logically called 'Logical Partitions?' -- well that's because that term is already used down in the cellar of Physical Volume Management.. and we would not want to get confused using that term again. An MBR disk can have four and only four Primary Partitions, after that you cannot create any more.

So planning ahead you can use one of your remaining Primary partition slots to make a special 'Extended' partition.. which is never actually addressed.. except as a pointer to a chain of 'Logical Partitions' underneath... these Logical Partitions (have absolutely nothing to do with Virtual Volume "Logical Partitions").

Sooo.. waay up Topside..on top of Virtual Volumes made from Volume Groups.. we call them 'Logical Volumes'.. ironic.. irritating.. and confusing. (they're [partitions] for cotton pick'n sake..! )

Let me state that again...

A Virtual Volume (which "really" is a Volume Group) is called a Logical Volume.. grrr.

The tools for performing this magic are exceedingly simple.. but hard to remember until you master their names.. and reasons for the choice of their names.. even if that reason is rather obscure and never really discussed.

First Physical Extents (the building blocks) are "made" using the pgcreate tool. (why isn't it called pecreate? I have no idea.. grrr)

Then the Volume Groups (the virtual hard drives or "volumes") are "made" using the vgcreate tool.

Finally the Logical Volumes (the virtual hard drive "partitions") are "made" using the lvcreate tool.

1. pgcreate - "pe create" - makes virtual "leggos" or "virtual disk storage blocks"
2. vgcreate - "vhdd create" - makes "virtual hard drives"
3. lvcreate - "lpart create" - makes "virtual (logical) partitions"

Each tool has a corresponding sister tool called "xx-display" to inspect the results and keep track of the "Virtual environment"

1. pgdisplay - "pe display"
2. vgdisplay - "vhdd display"
3. lvdisplay - "lpart display"

Now once these "virtual volume - (logical) partitions" are created they can be accessed from the /dev or /dev/mapper points just like physical hardware.

And the same tools used for creating a file system can be used on logical volumes to create file systems. Mkfs could then be used to lay down a fresh xfs file system and will be handled by the kernel device driver for xfs just like a physical hardware device file system.

# mkfs -t xfs /dev/vg-group1/lv-volume1

(think of it "like" )

# mkfs -t xfs /dev/vhdd1/lp1


# mkfs -t xfs /dev/vg1/lv1

Then the mount command or the /etc/fstab file can be used to attach the new device and connect it to a mount point on the current file system.

Anything that happens below the "virtual volume" or "volume group" layer.. will be hidden or transparent to the activities of the overlaid pavement of the logical volume (aka the virtual volume 'logical partition' ) and file system.. this is the 'Enterprise quality feature,, which desktop users can also use'

If we need to add more space to a full Logical Volume file system.. we can simply add a hard disk, carve it up into more PEs with the pgcreate command and add those to the volume group, then use an LV tool called "lvextend" to make the partition "bigger" while the file system is being used.. and without backing up the contents and resizing the file system and then restoring the files (a lot of maintenance down time).

Likewise, if we need to "remove" or "replace" a disk (perhaps it failed, is failing or S.M.A.R.T. tells us its expected to fail or some other reason) we can use pvmove ( it stands for 'physical volume Move' why not PEmove ? I have no idea...), to clean out all of the PEs from one disk or partition that is part of a volume group.. without notifying the upper layers, like the LV or file system.. or user.. this "frees up" the physical hard disk or partition and we can take it out of service and replace it. All while the system is running.

The major difference between 'Enterprise' and 'Desktop' is really in the details of whether 'While the system is running' means 'hot' as in 'Live to the world' or 'warm' as in 'Being used but can be rebooted to perform some quick task then back to service'. The game is to minimize system unavailability.

MDRAID or Multidisk RAID (aka software raid) or its likewise drivers can use LVs just like real physical disks or physical partitions to create fast RAID 0 or slow but resilient fail safe RAID 1 drives or anything in between. But they really don't require LVM.

LVM can also do nice things like make Copy on Write or Snapshot images possible.. but those are not fundamental reasons or purposes for LVM to exist.

Including obscure things like ( MDRAID, CoW, journaled file systems et. al. while 'sexy' ) in a newbie introduction simply flys over the important details of LVM and serves to confuse newbies about a very important tool that has become essential in daily life.

The terminology is a quagmire of a historical word swamp and does nothing to make it understandable.


Czur, project review for 2016

I don't work for Czurtek, but I contributed to their Indiegogo project and received an ET16 document or book scanner early in 2016. Here is a few thoughts on how my perception of this project has changed.

First, I don't want to make this a long article, but wanted to pull together my shifting thoughts on this project here on January 15, 2017.

When I started with the scanner I had thoughts it would be similar to a Aztec Booksnap or Fujitsu 4220c or somewhere inbetween. I was wrong.

It does scanning well and produces a collection of scan images [per session] and then provides a [work table] on which you can perform clean-up of the images individually or in bulk, then choose to create OCR - Optical Character Recognition, and bind the touched up images and the OCR information into a Searchable PDF file for archiving or reading later.

That was the dream, but it fell short in a few areas.

The Desktop PC for Windows software had some challenges in communicating what it could do, how you should use it, and in how to update it.

The Desktop PC for MacOSX software was not released and its been quite a while, but a new pre-release demo now exists which appears very similar and very good.. but so far also lacks good documentation.

The Desktop PC for Linux (if it is planned) has also not been sighted.

Over the last year I have had good luck at contacting support and they have responded with a private Twain32 driver for Windows, and an updated Twain32 driver for Windows (I have yet to review).

They have also shown me an SDK for developing programs on Windows that by-pass the Desktop software and scan images direct to the file system.

On my own I've come to understand that they are not very precise on releasing details about the chipsets or optics used in the hardware, and my guess is that this is due to Non-Disclosure Agreements with iCatchtek, or SunPlus which seem to do a good job of keeping their datasheets and programming guides off the Internet.

But on the good side, it seems the resolution and optics are so good as to put precisely considering and managing those details are not as important as they once were. From an engineering or obsessive detail persepective this can be frustrating.. but in the end really not that important. Any optical barrel distortion effects are confined to such a small field of view its just not that important.

This is also not that much different from the level of detail available for the Apple iPhone.

Its important to understand too that USB 2.0 or USB 3.0 has limits in the available bandwidth for capturing an image and moving it over to the PC across USB. Too long a delay is simply not acceptable, so often MJPEG or Motion JPEG of a high resolution is used to moved scan images across the USB cable. YUY2 (uncompressed raw) is available, but only at lower resolutions.

The controls on the desktop software are more or less like the [Auto] or coarse level controls available on 'Point and Shoot' pocket Cameras, demanding few specifics. These limited Profiles set the scanner camera, lights and image capture chip up based on the general question [what are we scanning today?] by way of -- 1. Is it a Full Color Photo? 2. Is it Color Line Art? 3. Is it a B&W Photo? 4. Is it Simple B&W Text?

Considerations like Dots Per Inch, Filters for Moire, and other things are left to "post processing" after the image is acquired.

On the PC the post processing is carried out using the OpenCV libraries.

As best I can tell the OpenCV libraries for the PC running Windows, use the DirectShow interface to create a FilterGraph, which is an object that continuously grabs frames from the scanner and passes them into a Null render object. On its was to Null however is a general purpose [grabber] object which can copy a single frame and place it in a memory buffer setup by OpenCV to receive the image, then any of the functions in OpenCV can be used to "Post process" the image in preview to optimize what will be scanned and captured then post processed into an image file on the PC.

Since the original scanned image is also available, before processing it can be saved too. This is as close to raw as we can get for now.. but its really not that important.

The major features touted for "this" scanner are the ability to "automatically" straighten or flatten the image, based on the additional ability to take a lower resolution scan with Laser beam guides. The high-res image and the low-res image are then "stretched" in OpenCV to mask and correct the 3D distortion from bending or folding until the Laser bean guidelines are straight. Quite a marvelous item in one combined product if all goes well. Some people call this "De-warping" the image.

 I am not sure how, but original images can also seemingly be re-examined and re-post processed, which suggest the lo-res image may be embedded into the original hi-res image file for possible later use.. but I could be wrong and just not understand the system.

The long term goal however is to make scanning and binding into an electronic document or eBook (in the form of a PDF) easy and without much thought. I think this would have worked.. if details about how it was handling the images were provided, and demos included clear instructions from A to Z on how to do this.

While certainly possible on Windows with the Desktop PC software, until recently it hasn't even been possible on the MacOS with the Desktop PC software.

The cloud solution seems to have trouble that users outside China experience as long unacceptable delays requiring an always connect state with the Internet.

The scanner is partially UVC compatible, meaning default programs on Windows, MacOS and Linux identify it, load their drivers and will use it for taking pictures, but generally low resolution pictures, and the default programs do not 1. provide fine control over the image contrast and brightness or colors 2. provide 2D straightening or 3D flattening 3. provide automatic OCR 4. provide a work table for touch up or binding into a PDF document.

In summary my perspective and expectations have changed over the year.

This is an Amazing product, still in development, and will be a great value when it is complete. And the renewed release of new firmware and Desktop software, the release of Desktop software for the Mac, release of Twain32 and release of a second generation of Twain32 driver plus an SDK that includes the ability to 2D straighten and 3D flatten.. all point to a very bright future for this project and product -- but -- it is not finished!

The biggest challenges are the "language" barrier and lack of sufficient documentation to not only "achieve" document scanning and binding.. but that guides and teaches.. and makes you comfortable with it.. are its biggest challenges today.

I regularly receive a lot of offline commentary and inquiry about this scanner and requests for my opinions.. but I always preface those conversations with 'I do not work for Czurtek..' so I cannot and do not speak for them.

I can say I wouldn't mind serving in some way to help with the documentation, but I think the language barrier flows two-ways.. I do not think they always understand what I am saying when suggesting small improvements that would make a 'big' difference in how people perceive their product.

Luckily I have a full time job, so the only benefit I get from making videos or answering questions about the Czur ET16 is when I learn something for myself or someone politely says a video or article helped them. And for that I thank anyone who happens to read these articles..it motivates me to keep going and not give up on the ET16..and sometimes I do feel like giving up.

But with patience and time, I can see its potential.. and that really there is nothing currently on the market like it. Its super simple, its complete, the results are really good.. and its still being improved.

Usually when something comes to market, all development stops and its frozen in feature time.

This scanner is not.. its improving and its getting better.


Czur, (beta) Mac OSX scanning software

This is a demo of the Czurtek scanning software for the Apple Mac on my MacBook Pro 2012. Its really good already and can only get better.


Programming, windows usb devices

I feel it useful to document by diary/journal when learning something new. In the confusion that exists before consolidating memories are "perspectives" born of a time when I did not know how things actually worked. In that narrrow twilight zone of understanding are common tropes or "pathways" that other people may travel.. perhaps because of a common heritage of prior experience. -- by documenting the misconceptions and "wrong headed" failures.. I hope other people can enjoy the "trip" and arrive at their personal destination (or similar understanding) that much quicker.

USB comes in three versions or "flavors" 1.0, 2.0, and 3.0 generally defined by their speed or data rate, but also defined by the hardware chipsets that enable the ports that physical devices are plugged into a computer.

The wire "signal" is a complex "balanced" overlapping arrangement of virtual bit trains and virtual "frame sets" which indicate chunks of information. Embedded in this stream are timing and data information and multiplexed "purposes" as there are co-existing "different" [types] of virtual pipes with differing behavior. For example, some traffic is "prioritized" over other traffic, such as for pseudo CPU "interrupt" driver processing service. Other types are for "lossy" communication in which dropped packets of data in exchange for "realtime" or "current" data is acceptable (think a telephone voice call where 'crackle or noise' is acceptable as long as the voice is pseudo-realtime and not delayed). Still other types are for absolutely "reliable" traffic even if it is delayed.

Quite a bit of the wire protocol is encapsulated inside the silicon "chips" which are embedded into the computer motherboard or inside the external hubs and usb devices themselves.

These are connected together with USB cabling to provide service.

There are standards for USB cabling and different capabilites with each type of cabling. But I won't get into those feature here.. since I'm more interested in learning to write a program or driver that opens communications and actually makes use of a usb device.

Within Microsoft Windows is an arrangement between user application and kernel applications where by the User Mode and Kernel Mode programs are written using two different sets of Headers and Libraries. Each has their own set of APIs (application programming interfaces).

For the most part these APIs are C/C++ compatible and most User Mode or Kernel Model programs are written in C or C++. Then compiled using either the SDK for User Mode or DDK for Kernel Mode. Each of these "Development Kits" contains their own "complier" customized and setup to reference the correct set of Headers and Libraries at compile time.

Traditionally Visual Studio was used to create User mode programs and Visual Studio was used to possibly edit Kernel mode programs, however when it came time to compile.. Visual Studio would use its complier to compile User mode programs and Kernel mode programs would have to be compiled manually with the DDK compiler.. this has changed back and forth between Visual Studio for both or separately for User mode programs only.. it generally depends on the era and which Visual Studio product is used.

Windows also includes a general purpose "database" which the User Mode or Kernel Mode programs can access at runtime called the "Registry". Historically evolved from a collection of [INI] files into a unified set of binary files to gain speed of access and to allow the database to grow much larger and faster than an ordinary text file based database would make possible. It also allows a certain obscurity or ofuscation (opaqueness) to the database which was later given.. even an access permission control system to prevent certain entries from being accessed or changed without the proper user permissions. All of this means even the User Mode and Kernel Mode programs have parts of their APIs dedicated to accessing the registry, which they access to make use of dynamic runtime libraries and COM/DCOM runtime objects.

All of that is history and window dressing for the main stage of developing application to access USB devices.

In the early days of windows, even in DOS, there were no USB "drivers". Each program had to initate the USB port and manage the abstracted protocol features the chipsets provided to communicated with USB endpoints which represented devices connected by USB cable.

In time however "Kernel mini-port" drivers were created to manage and "share" access first to USB Host controllers, then USB hubs, and finally USB devices. As this "Stack" of drivers came into being Microsoft sought to make driver writing simpler and faster, as well as less error prone.

So the evolution from "mini-port" drivers was to "device driver frameworks" to assist driver writers in creating drivers based on examples or "templates" which guided them in making similar drivers that had similar features.. like debugging or other features.. so that they could all be treated similarly by the overall operating sytem. -- in particular the jump from 9x (VxD) and NT kernel device drivers, was to the WDM - Windows Device Model kernel mode device drivers. (soon to be replaced by the WDF- Windows Driver Framework)... Which led to frameworks for creating all kinds of device drivers.

Specific to USB however was the establishment of the [winusb.sys] device driver which took over much of the job of older or vendor specific device drivers and co-operated with the central operating system kernel to share and manage the USB [bus] as a common "thing" that might be used simultaneously by many programs at the kernel and user mode levels.

USB is a "shared" communications medium, even if it may appear at times to be a cascading set of point-to-point connections. At any one time the total bandwidth of a singular USB port on the computer may be prioritized or divided out among many USB devices on the virtual "bus" shared by all the connected devices.

So once the WDM framework made device driver writing easier and the shared USB bus management made using multiple USB devices simultaneously possible.

Microsoft set about co-operating with the USB Special Interests Group (SIG) or USB Forum, which had shared interests in "standardizing" groups of similar USB devices and coming up with agreed upon minimal protocols to use these groups of USB devices. They were called USB "classes".. not in the sense that they were abstract Programming classes, but that they shared a common set of designs and features which could be probed and expected to exist, and thus similar drivers with "vendor specific" deviances or functions could be accessed.

The win for Microsoft was they could include [in-the-box] with a new version of Windows both a winusb.sys driver for oddball USB devices that were unique, and "Class" device drivers which would provide certain minimal services for "recognized" USB devices.

The USB devices were "recognized" by a process called discovery or "Plug and Play" detection and device driver loading.

USB devices generally have an established protocol where "descriptors" or "strings" of information can be retreived from a USB device after it is assigned an address on the USB bus.

The assigning of the USB address is a base function of the USB bus driver after a USB device is plugged in and powered up.
 Once  the "Strings" are retrieved a search of the strings returns a PCI Vendor ID "like" unique finger print which identifies the device as a possible member of a device class, and a specific vendors product. Depending on the device drivers registered in the Windows Registry or the INF flat file database an appropriate driver or driver "set" of device driver files will be loaded and start execution on behalf of the device in the Windows memory space, in general these are Kernel mode device drivers of the WDM type, a USB bus manager, device driver and class device driver.

The end result is a User mode program can start up and probe the system device list for a target which represents the USB device it needs to peform its function. And by sending IRB (Input/Output Request Blocks) to the Class driver communicate with the USB device.

IRBs are very low-level however and generally a Class driver has a predefined set of headers and library or APIs for communicating with the USB device it manages.. abstracting away (or simplifying) the level of effort required to write an application that uses the USB device.

The really hard part (I think) about all of this is knowing the history of what came before, so that not only you have perspective, but can also "filter" or "ignore" whispers of irrelevant data .. such as information about a VxD, or about a long deprecated protocol like VfW - Video for Windows, which trying to get a foot hold in the current era can distract you.. or prevent you from choosing the most appropriate API to use for the Class driver your interested in probing and using.

There are "models" within "models" describing both "abstract" representations (or "perfect" idealized models of how things work) and then "real" representations of actual devices which have [some] of the chracteristics of the ideal "model" + plus "extentions" or features that the model does not address, which you then have to figure out how to address. Some Class drivers build in (vendor specific) methods to access these other features.. but some do not.. in which case you may have to consider writing a mini-port driver to co-operate with a Class driver (an extension "plugin") .

The USB forum (or SIG) also describes many of these same things in similar but not identical terms, so that an abstract model in Microsoft documentation may not line up with the description  of the same thing in a generic USB forum Spec written with the idea of serving multiple purposes beyond Windows programming.. such as programming for Linux or Apple OS products.

The USB Specs are also [very] hard to read as the knowledge base of the authors is both jargon filled, presumptive and lacking in details, and written "backwards" or "upside down" in that they are written more like legal contracts preceeding object of discussion with adjectives or descriptions "before" refering to what they are actually talking about. In Germaic descendent languages like English, it can be very hard to [overlook] or [ignore] the fact that the document appears to have no purpose or subject matter, until "after" the discourse on the descriptions and actions.. leaving motivation to "finish" reading the Spec document..very difficult to find. (I believe) certain speakers in other languages will have an easier time reading the documents as I have seen similar "speech" patterns in other languages.  -- once "consumed" however Germaic readers can generally "re-organize" the material almost subconsciously within their mind.. much as when translating a foreign language internally.. and its easy to "forget" how hard the first reading of the material was to understand.. a certain amount of "faith" or experience in readin RFC or Spec documents may make this easier.. but for the inexperienced or "uninitated" newbie.. this can be a very difficult barrier to overcome.

In the perfect Microsoft Windows programming world. A USB device will have a high level, high functioning Class driver and the end user User Mode Application will have an available "framework" specific to their task.. such as DirectX Direct Show for Video, and the end user will not have to write a device driver, or understand many of the details of how the actual silicon or the device performs its task.. it just appears to do so..  like "magic"

A further "funny" detail is that you should not loose sight of the fact that a USB bus is a negotiation between two independent devices arbiting over communcations and services. The USB device is itself its own computer, with its own operating system (however simple that may be) and thus has its own programming. Often called "firmware", it can have bugs, it can change, and features that were available in one version of the firmware can change or be removed in the next.

Generally the USB descriptors are used as the IUnknown or COM type object feature of "discovery" and "enumeration" or list of available features.

It is through the USB descriptors a USB device provides about itself that Class drivers can setup and expose features of the device to the operating system user mode programs.

In the inital release of a hardware USB device, the firmware may not be 100 percent complete. And the USB descriptors may describe features or functions that are not quite bug free, or even implemented yet.


CHM, Help 1.x files missing Contents Index Search

CHM - Compiled HTML for Microsoft files can appear damaged or unreadable. But they are not, they are often distributed in multiple "pieces" rather than a "singular" name.chm file in older Microsoft PDK, SDK and DDK documentation. (accidentally) moving only the .chm file can result in missing functions when opening the .chm file for viewing.

For example:

The image has been deliberately "blurred" to protect the rights of the publisher. However the controls and navigation pane to the Left have not been "blurred".

You will notice that The usual [Contents][Index][Search] tabs are "missing" from the tab menu.

Typing "anything" into the [ Type in the keyword to find: ] then [List Topics] is equally useless:

The word "untitled" will be returned over and over in the topics list, and link nowhere.

The problem in this case is the directory contains only "one" file:


But the original document directory in the original install folder contained:

C:\sdk documentation>

The "missing" .chi file is in fact an index file, and was intended to "separate" changable information from static information when distributing the files on CDROM. It is described somewhat in this old link to Microsoft HTML Help

When you recover or "move" the .chi file into the same directory as the .chm file it (automatically) indexes "on the fly" and the results are as expected (minus the "blurring" effect I added to the image).

That is the "expected" navigation controls on the Left return and are available again.

Also the "existence" of the .chi file causes a "new" file to be created when the .chm file is opened. In fact the procedure is that as soon as the .chm file is opened a compatible .chi file is searched for in the current directory and then a new .chw file is created if the directory is "writable" - this is described as a combined kwords file.. or "chw"ords file. "k" is often used to mean "Index" file in other CHM contexts.. so this could mean "Compile Help Index Words" or "Compile Help Key Words" to support the Index and Search tabs (without the extra "k").

Its further confusing that [some] in fact [many] files compiled in more recent years, especially by third party software programmers, include the index (inside) the single .chm file.

So those .chm file behave as if the .chi file were included in the current location, but are in fact "standalone".

This change in behavior really makes troubleshooting older documentation harder, especially since Compiled Help documentation (about the CHM format) is somewhat rare.. and the change in Help document formats and ramping up of Security blockades to disable and deprecate older Help file formats. -- there simply isn't a lot of care taken to "learn from the past" and "document for the future".

Note: further online research led to the tacky "political" situation between "Public" chm files and "Private" Internal Microsoft MSDN formats. HTML Help versions 1.x 2.x and 3 or 4,5 were on a very rocky road at the end of the last Century (1999-2000) and after announcing and "using internally" a new version of  CHM they basically cancelled and did not deliver. The result was a split or division between versions of CHM 1.x and 2.x (which was never released outside Microsoft). The MSDN Library released on CDROM would use this new "splitted" CHM/CHI/CHW version.. and apparently without formal notice the SDK, DDK documentation.. and perhaps Microsoft Press CDROMs.. but it was never formally documented.

Some people took to reverse engineering the new pseudo-unoffical/official format and came up with tools and "ways" of adding their own .CHM documents to the MSDN versions.. before the later versions of CHM.. so the internal strife were recognized and "patterned" around.. which is nothing new for this era with the company.

The CHM format appears abadonned, but maintained in more and more projects, outside the oversight of Microsoft.. and as they do not maintain much of a corporate history.. perhaps the less involvement and further they get from it the better. But the standard for CHM while not open, or well documented lives on. With only .MD or HTML markup showing a hint at replacing it on GitHub.