Czur, the SUNPLUS Burn file format

The ET16 scanner is based on the Sunplus or iCatchtek spca6330 image processing board. Many vendors use the same hardware components. The EKEN ("eee-ken") action camera also uses the spca6330 board and a Windows PC hosted firmware flasher called the FRM ("firmware resource manager") to perform ISP ("in system programming"). The FRM can also produce Card ISP files ("insertable memory card in system programming files") . The Sunplus BRN (""sunplus burn files") are memory "Card ISP" files

These can be used to update firmware in production devices capable of being placed in RSV mode ("read save verify" mode).

Embedded hardware comes with NAND, eMMC or SPI accessible EEPROM memory. This is where the firmware is stored. Typically its divided up into chunks or "partitions" which are accessed by the bootloader at the start of memory when start up commences.

If a generic operating system like Linux is used the path of least resistance is to format some of the partitions as "virtual disk drives" with a well known file system such as ext2, or a compressed file system like cram or squashfs. In that way a generic monolithic program like Busybox can act as many parts of the operating system in a single binary to save space while exposing a common and familar (and scriptable) toolkit and programming interface.

[- then again on second thought nothing precludes "burning" as part of a partition a small file which contains a partition "image" of a filesystem itelf, like a fat16.img file.. -]

If a more conservative and deterministic (aka reliable) or 'time critical' operating system is used like eCOS, RTOS or ThreadX then the memory chunks or 'partitions' may simply be "mapped" regions of storage accessed by the programs themselves without any file system.

The SUNPLUS BURN FILE format appears to be more of the latter type.

There appear to be three partitions, two general "resource" partitions in which general purpose tool binaries are placed, which can be anything from graphical image files to font files or actual programs.

And a third partition dedicated to "default firmware", which some might call a "bootstrap" or "boot loader" except that in this context its actually more like a combination bootloader and kernel.

Resources from the other two partitions are accessed by the "kernel" using a known offset in persistent memory devices always present on these "known" hardware component boards.

The exact architecture of the file formats is not known, however they are probably a type of ELF file format, and given the processor is likely an ARM A9 it is probably for a 32 Bit ARM processor architecture.

Its more common these days with multi-core or multi-processor chips to run two entirely "different" operating systems on each core.. one dedicated to a "time critical" -- real-time -- operating system like ThreadX, and the other to a general purpose (or easier for 'humans') to script or program [user-interface] operating system like "linux".. this is called [a]symmetric multi-processing since they do not neccessarily run in synchrony and rendevouz ("link-up") by passing messages back and forth through established methods in their operating systems.. its more like the worlds "smallest" internet of computers. - This does not appear to be the case with the ET16 scanner.. if it is running two copies of an operating system, they may be the same operating system. (I base this on observing both partitions appear to simply contain binary blobs and not a recognizable 'linux' or 'dos-like' file system).

While it is possible to use the FRM tool on a production device in RSV mode to "download" or "retrieve" the actual burned firmware and make a "backup" before flashing new firmware, or for further development and inspection.. I have not done this with the ET16 scanner.

When the binary "blobs" are retrieved in this fashion they come back and are saved on typical Windows file folders. And can be further examined.

My purpose is not so much to "reverse engineer" as to simply understand in "general" terms how the ET16 actually works.. and so much of this is from inference and observation than actual "hacking".

Besides, the Sunplus Burn File (SPHOST.BRN) or "System Programming for Host Burning" file is already provided as a "firmware update" for the ET16 as a download from a website.

"Unpacking" a [Sunplus Burn File] should be a matter of looking at each "partition" inside the singular Burn file (unfortunately not a file system you could mount) and extracting the individual binary ELF format programs. - but then we'd have to stop, since unless you had a sufficient debugger or hardware development board. The programs would not run on a PC.. perhaps inside a virtual machine like QEMU, or debugger like IDA Pro.. but that would pretty much be end of story.

One gentle man has used Python to create a few scripts to extract the ELF binaries from a Sunplus Burn File

Reversing, Exploiting und Patching an Action Camera

Martin's GitHub for SPCA-FUN python scripts

One thing that came to mind was as a 32 ELF the byte-ordering might be reversed, and this might be the case in the final Burn file as well.

I hope this satisfies a few peoples curiosity.

p.s.. The EKEN (pronounced "eee-ken") action camera uses the "resource" partitions to store "parameters" or settings which the other programs access and use while running. By tinkering with these settings they have enhanced or modifed features and behaviors of their action cameras. This is a dangerous thing to do. And it does lead to "bricking" or making their $300 to $400 cameras useless.. but explains much of the interest in "hacking" their devices.

p.p.s. Sunplus is a collection of companies in China and has a large and deep history with good documentation and a thorough patent portfolio. Not all of their documents have an English language counter part. And many of their documents are proprietary and confidential. The few public samples I have seen from many years ago are excellent and quite detailed. (I am under no NDA). Sunplus does responsible releases of public GNU code that they use, but do not include the specific binaries and scripts they are not beholding to release. -- That said reverse engineering to bring much of the hardware based on their technology under support in Linux is always ongoing and appears to take place mostly in Europe. -- From I can see Czurtek is probably under NDA and cannot release many specifics about the actual EVK software development code and hardware interfaces. They have however built a rich API on top of the proprietary code and expose what appears to be a very useful API of their own.


HP Folio13, How to Fix the WiFi button

Hewlett Packard made a laptop called the HP Folio13 a few years ago. The WiFi would work for a while and then appear to stop working (even if the WiFi keyboard key was On). This is how to fix that. Basically "only" the HP Connection Manager could turn a Secondary "Airplane Mode" switch off and on inside the laptop. Later releases of the drivers on the Laptop Support page did not include the older HP Connection Manager tool. -- If Airplane mode was turned on, and the drivers upgraded, you no longer had a way to turn the Airplane mode off, or re-enable the WiFi internal switch.

Demo of Flipping the internal WiFi Airplane Mode Switch

This internal switch is [independent] of the keyboard WiFi switch, which will remain On at all times whether the Airplane mode is turned on or off. It is also "counter" intuitive, but Airplane mode ON, means the internal WiFi Airplane mode switch is turned OFF. And the two are not "Exclusive".

You can turn the Airplane mode ON, and then manually select [only] the internal WiFi Airplane mode switch independent of the internal Bluetooth Airplane mode switch. (AND) the internal WiFi Airplane mode switch "is NOT" labeled as a separate switch.. but it behaves as a separate switch.

The proof of this, is that the ERROR condition when using the Microsoft Windows Troubleshooting tool indicates the WiFi Radio is OFF and instructs you to turn it on, to resolve the problem. Which is NOT possible, because it is already ON.. toggling it back and forth ( regardless of the lighted LED state ) has no effect.

The keyboard WiFi switch has a white LED in the upper right corner of the keycap which remains ON at all times while the internal WiFi Airplane mode switch is "flipped" back and forth using the HP Connection Manager tool.

When the keyboard button is actually Turned OFF the HP Connection Manager displays a [Totally] different "kind" of OFF. ( this is "Real" Off, not "Airplane" Off )

Airplane mode OFF is much more subtle and "mis-leading" when the WiFi button is ON but the internal WiFi Airplane mode is OFF - it merely says "Off" and not "Disabled by Wireless button"

Technically, from my investigations it is a (problem with the Bluetooth sharing the WiFi button) in the programming logic for Airplane mode. You can switch WiFi or Bluetooth Off or On "while" Airplane mode is Off or On. This drums up the possible combinations to at least 6 and possibly 9 "modes" which the display interface does not accomodate. And while they "tried" to cover all combinations.. the user interface failed to do so.. and the Intel Connection Manager doesn't even try.. it just fails. -- dumbing it down.. just make sure to turn the internal WiFi Airplane mode switch OFF or On from inside the HP Connection Manager.. there is no other place you can control it.

[AND] don't try to control the Bluetooth state from the HP Connection Manager, only use the keyboard button. -- This is fundamentally what broke everything.

Upgrading the driver from the HP Support page for this laptop installs a "new" Intel Connection Manager, which cannot disable or manipulate the internal WiFi Airplane mode switch on this laptop.

You can still uninstall the Intel Connection manager, then use Windows File Explorer to browse to the C:\swsetup directory and find the specific HP spXXXX package directory and a subdirectory under the the new Intel drivers.. (avoid) installing from the top level directory that would also install the Intel Connection Manager, instead go to the drivers only directory and look for [ dpinst64 ] and double click that .. it will [ONLY] install the updated Intel WiFi drivers with NO Intel Connection Manager. Then find a copy of the older HP Connection Manager installer and install that to [regain] control over the internal WiFi Airplane mode switch.

Beware!! the fully opened HP Connection Manager cannot be used to turn the internal WiFi Airplane mode switch off and on. [Only] the right-click system tray icon (round blue ball) and going straight to the WiFi switch under the Airplane mode menu actually works.


Kdenlive, a replacement for FinalCut Pro or Adobe Premiere

Opensource on Windows usually doesn't blend well. But Kdenlive is a suprisingly functional and smooth NLE - Non-Linear Video Editor that just got ported to Windows and MacOS. Free as in Beer it has all the features you would expect in the old FinalCut Pro, Adobe Premiere or Sony Vegas Movie Studio products. Multi-track, themeable and stylable its also easy to adapt the editor look to something more comfortable. kdenlive.org

It has a ton of transitions, color correction, and video and audio effects. Its built upon several ultra-portable opensource project foundations which doesn't tie it to one PC platform. Originating on KDE on Linux, it runs just as well on many vintages of Microsoft Windows and Apple MacOSX.

If your in the market for a video clip editing tool of the calibre of FinalCut Pro or Premiere, definitely give it a shot. On Windows its still so new there isn't even an installer. Its just a smallish Zip file to download and extract, and a few ffmpeg files to copy into its core program folder, and a presets folder to copy from the ffmpeg distro into its core program folder and then simply double click the kdenlive.exe - there is a warning box that currently flys up and suggests some missing icons and add-on tools.. but they are totally unnecessary. The default windows MFC control icons are both familar and makes it feel much more "native" to the Windows operating system.. and I presume the same would be on MacOS.

The Linuxscoop YouTube Channel reportedly uses it to edit some of its videos (if so):  then take a look at that channel to see if its what you've been looking for -

I would caution (a) it's "Zippy" as in fast, and (b) its not a "Storyboard" editor like Windows Movie Maker or iMovie.. those are [cut and paste] editors that let you draw a story like a train with box cars.

A Non-Linear Editor, gets its name (NLE) from the ancient artform of literally cutting film into strips called "clips" and then resequencing and splicing them together (either) one after another or even (on top) of each other to "composite" an entirely new visual scene.. this was the way Special-FX were done in Hollywood back before there were "computers" and even how the first Star Wars movie was made.

As an NLE with infinite [tracks] both above and below, Kdenlive.. can composite.

[Note: I got a an out of bandwidth question about setup. It does (Not) have a slick installer yet, its still just a Zip file download. After downloading on Windows, extract it. And download the ffmpeg Zip file which is linked from the same page. Extract ffmpeg, inside that folder are ffmpeg files, copy those over straight into the Kdenlive folder, [and] copy the ffmpeg "Presets" folder over into the Kdenlive folder. Then you can double-click on the > Kdenlive.exe and everything starts up and runs just fine. -- understand Windows isn't their primary focus.. it just happens to run extra sweet on Windows.. enjoy the Candy.] - {You add Tracks from [File] > [New] > mid-page > "Video tracks" or "Audio tracks"}


Czur, How to Scan Books

Scanning books begins with capturing images using an image scanner. It also requires lighting and an environment that presents the pages one after another. When complete the operator is left with a stack of images that require processing.

During processing the images may also be read with character recognition software to create  keyword content that represents the image content. This means images can be inspected and recognized as containing language text which can be used to replace the image content in a new image-less document format, or used as a keyword map to find coordinates in the image of the page.

Finally images are bound into an electronic doc, like an ebook.

These are the steps to transforming a real book into an ebook.

A couple things to consider:

Choosing an image scanner and a lighting environment and scanning hundreds of pages per session can be a time consuming and difficult task.

Choices for the scanner are many but may be limited by available budget.

The Czur ET16 scanner is one scanner that costs less than many images scanners. It includes a lighting system, hand/foot controls to remote trigger an image scan (and autoscan which can detect when a page has been turned and automatically trigger a new image scan) and includes software to perform post processing, optical character recognition and binding into portable document format (PDF) ebooks.

It arrives as a kit that requires some assembly. The software is downloaded from the internet and installed to a Windows PC. During installation the software is activated using an activation key attached to the bottom of the scanner. The Windows PC is connected to the scanner by a USB cable and turned on. The Windows PC recognizes the scanner and completes device setup.

Scanning a book can take considerable time. It may require more than one session. Each session has an associated date and time when it began. The software tracks the date and time when each session begins by creating a new folder on the file system for each session labeled with the date and time the session began. 

To scan a book one begins by setting aside a block of time in which they are less likely to be interrupted. The Windows PC and scanner are moved to a darkened room with lighting less likely to interfere with the light provided by the scanner for capturing images. The background mat is placed below the imaging sensor and lined up with the edges of the scanner.

The Windows PC is relieved of extra running processes and the Czur software is started. A USB cable is connected between the Windows PC and the scanner. A choice is made to use the hand or foot switch  ( or to rely on autoscan ) and if necessary is attached to the scanner. The scanner is switched on.

After choosing to Scan rather than to Present.

The Windows PC running the Czur software will display the 'work table' for performing post processing, optical character recognition and binding. Before these tasks can be initiated a Scanning session must be performed.

The upper left menu [Scan] button is selected and a 'sub application' is started that searches for a scanner connected by USB cable to the Windows PC. A live 'field of view' is presented in the center of the screen which shows what the scanner can see. 

The right shows the current directory listing of the current 'date and time' session folder. 

The book to be scanned is placed in the scanners 'field of view'.

Scanning a book cover will not require holding it in place. Scanning the interior pages of a book may require holding the book open using fingers or hands.  The scanner will automatically process an image with color drop-key to remove fingers or hands wearing colored drop-key covers while holding a book in place. Color drop-key removal will only be performed for fingers or hands found to the left, right or below the page content.

When a scanning session is complete the [X] in the Upper Right is clicked to close the 'sub application' this will bring all of the scanned images to the work table and populate the Right navigation panel with their names.

As each image was scanned it was copied to the PC and given a filename. As each filename arrives on the PC it is "processed" based on the "profile" that was selected for the scanning session. A 'pro-file' defines how to 'process-files' for a session.

A work table full of scan images can immediately be bound into a PDF or TIFF file by going to the Lower-Right and pressing [all] to 'Select all' images. Then going to the Upper-Left and pressing 'Archive' and selecting [as PDF] or [as TIFF]. A request for a destination and filename will appear and create the 'Archive' document.

PDF and TIFF files are 'multi-page' documents which can be viewed in a PDF viewer like the Chrome browser, or an Fax viewing program.

While the above description is the simplest workflow.

There are many variable settings, controls and tools which can be used at each step to attain greater control over the overall process and final output.

Global settings such as Scanned Image resolution, Image file format, dpi and Left/Right page splitting order are chosen from the Upper Right [Settings - Gear] icon.

Session settings such as Color Mode (a combination of Brightness, Contrast and Sharpness) are selected from within the Scan 'sub application' to the Upper-Right just before scanning. And they can be tweaked somewhat by disabling [auto] exposure and manually moving the slider control underneath the preview image.

Session settings such as picking a Profile (a combination of deskew, autocrop, dewarp, page-split or none) are selected from within the Scan 'sub application'  to the Lower-Right just before scanning. Tweaking is performed upon the image files on the work table after 'sub application' is closed.

When working on images on the work table, a single image can be tweaked at a time with the controls underneath the selected file image. Multiple images can be tweaked in batch sequence by selecting all the images making up a batch group and then prototyping the changes to be applied to all of the images in the batch group from the [Bulk] menu option.

Archiving allows creating PDF or TIFF files, or OCR doc file and optionally combining those with PDF files to create a variant of the standard PDF format. These PDF variant files can include a keyword index that maps to locations within the image pages in the PDF file. This makes the PDF image document keyword 'searchable'.

The end result of a scanning session is always a multi-page Archive PDF or TIFF. The scanned images from a session are removed to ensure space is available for the next session.


Czur, skinning with DuiLib

The Czur windows software has an easily modified user interface. I noticed the choice to use DuiLib when I went looking to see if I could change the keywords with Astrogrep. That revealed a simple xml text file with Chinese to English mappings. Then I got curious.

The "skin" directory contained a series of xml files and png files which looked very regular. Also the binaries included a dll library called duilib_ud.dll googling did not turn up much, but a very few hits from 2005 pointed towards a Chinese opensource project to re-use code from an effort to make user interface design simpler and better. Microsoft had released wpf - windows presentation foundation, but most felt it was too heavy and difficult to learn and that most people would not use it. Internally Microsoft was known to use simpler easier to use tools called "Window-less Controls - also" they did not release to the public. So one person Bjarke Viksoe  released an opensource project that emulated that "known but unobtainable" framework. The Chinese project DuiLib Group took this and built upon it. It was called DuiLib - Direct User Interface Library. It appears to be cross platform, adaptable to MacOSX, Linux and Windows.

DuiLib takes an XML file and a bunch of graphic files and creates controls and pastes them on a single Window which shares its window to the Window-less controls which users can interact with. Sometimes its promoted as a "window-less" user interface in that it doesn't exactly emulate the modal windows that a Microsoft Windows program might display. But instead produces "canvas" with controls layered on top. This makes it simpler in that its more like a webpage, and many of the hidden behaviors of more complete user interfaces are not available to confuse people who would not typically use them. So.. its rather like making "Pizza".. and everyone likes Pizza .. right?

So the idea of the "Skin" is just like a layered "Pizza Topping".

Hacking XML files in a text editor is not a lot of fun. Fortunately the team (called the DuiLib Group) also created a GitHub account for DuiLib Modify and released a toolkit with sample user interfaces, the duilib source code and a graphical GUI designer with a Visual Studio 2008 .sln (solution file).

That meant I could install Visual Studio 2008 and compile both the duilib library and the DuiLib Designer tool, and use that to Graphically study the Czur XML skin files.

The only problem that came up was that DuiLib is mostly documented in Chinese. And I speak English. A lot of the source code was in English, but the user interface controls were labeled in Chinese. I used a hand held visual translator to look at the labels on the screen and translated those into English by hand and compiled the DuiLib Designer as a binary release. Then I created an Application distribution project added to the VS2008 solution for DuiLib and created an install package.. Setup1.exe... and put that on the laptop attached to my Czur scanner and installed it.

It worked fairly well, though I do not have a project file with the Czur skin, DuiLib Designer can read the XML files and import all of the graphics and render the canvas panels on screen.

It is not perfect and I still have a lot to learn about the Designer. There are docs, in Chinese for it.. but I don't really plan to go as far as to create a Project. Mostly I just want to relabel controls and perhaps change the colors of a few elements.

I am not sure this could be done on an Apple Mac, and that software while close is not finished. But it might, there is no reason to think they did not follow the same design principles.

And it kind of points out this may be the direction they follow in bringing Czur to the Linux platform.

DuiLib is opensource, and its platform friendly.. not "exclusionary" or "clique-like". It has broad support and appeal for billions of people in China, and now maybe for the rest of the world. Learning about the Czur skins seems like time well spent.


Czur, bookscanner Language translations

Czur scanner keeps delivering surprises. I found the user interface is based on a framework with a skinnable template. Inside the skin folder was a single Chinese to English mapping file. So it was easy to change the odd or difficult to understand word choices into something better. - since then I've recomposed most of the Scan dialog window and Publish and Bulk edit user interface windows using the translate file as a personal notebook to record my discoveries. Its gone pretty good.

Soon I hope to condense what I've learned into a few short video tutorials in English for anyone interested.

My thoughts are both jumbled and excited, but they are also getting simpler.. distilling what exactly is or will be possible and what will not.

The scan dialog reduced down to just Color Modes and Profiles. The scan resolutions are preset in the application itself, but also limited by the defined capture modes pre-programed into the optical sensor device and its onboard embedded computer. In general there is a location to make a single choice for the default resolution, which is pretty good. Custom choices are possible, but the default is dense enough that other than using it for a microscope it should be sufficient. Lighting, Gain, Contrast and Brightness adjustments, also while possible are preset in the Color Modes. So when capturing they really are not a concern.

Once a single image is captured with a scan, it is uploaded across USB to the computer, where the chosen "Profile" for the session, continues post-process the image files into their "temporary presentation mode". But they carry all of the original captured image information in this form so they can be reverted back to original form and re-converted into other Color Modes or Profile types. What actually occurs in the "post processing" depends on which Profile was selected.

For example the one for single flat pages does far less, than the profile for double bound pages like those in a book. Where the single flat profile will straighten and crop and stop. The double profile will square the curve linear lines projected by the laser beams, then construct a map of the dual page surfaces and attempt to flatten the pages. Then that profile will automatically cut or "split" the single page into two separate files.. and depending on a global option as to which should come first in a stack.. Left Page then Right Page, or vice versa.. each is written out with an automatically generated name. Right Page then Left Page order is also a selectable global option.

After capturing scan images into these "temporary" session files, closing the Scan window takes one back to the worktable, in which the temporary files appear in the file list panel to the Right.

They must be individually or in 'Bulk' selected in order to perform a "Publish" operation.

The Publish menu choice then offers to (a) create a simple PDF (b) create an OCRd and indexed, therefore 'searchable' PDF (c) create only an OCRd searchable index or (d) create a lossless multipage TIFF file with or without compression.

Lossless is better for archival purposes. The original captured image is used to create the second generation lossless copy, but the degradation stops there.

If you attempt to close the worktable before Publishing, it will warn you that all of the session scan images will be wiped from your work space and will be subsequently unrecoverable.

On the worktable are tools to basically "tune" or "convert" choices made when the original reference scan images were captured. These can only be used on the current session scan images. While other previous session images, or other suitable images can be included and inserted into a current session document, they cannot be edited with the tools in this session.

A final menu option is for performing each "tuning" tools action on a group of selected scan images sequentially one after the other in a Batch mode. This is computationally intense and time consuming, but the batch function provides a scheduled job list and task schedule to monitor the situation.

Other document or ebook creation software packages should be fully capable of working with the session scan images and exported or saved TIFF files and go further in a more comprehensive archival documentation project.. for example if adding EXIF meta data, or sRGB, adobeRGB color space information is important for you... you can with supplemental software if your willing to purchase and learn how to use it. The Czur kit however will serve as ample low cost training wheels if a $500 or annual subscription to Adobe products is not your first inclination.

It is important to acknowledge that due to practical USB transfer speed, and most peoples patience, the scanner only provides access to JPEG compressed images of a relatively high resolution. Saving them as TIFF will insure they do not loose any more to generation loss.

The scanner is also a UVC compliant camera, and exports USB Endpoints, some which advertise YUY2 ... so 'lossless' images of lower resolutions are possible.. if that is a real need.

I have had a chance to preview the OSX version of the scanner software and it appeared to support a similar work flow.. which is very generic, well supported and well understood.

I have also had a chance to preview the Twain32 driver and it works well.

And I have seen, though not worked with a set of exported functions which would allow any program capable of Win32 COM calls to take control of the scanner over USB and perform most of the same operations as the provided application software.

Detractors are that there is a language barrier, and it is not a finished product or project. Firmware and application software is still being developed and released and its support base for other operating systems is still expanding. It is a USB video class device so there should be no limits.

But on the other hand it is extremely "open" to end users and Third party contribution and development, other than tiny and specific technical details and specifications of interest primarily only to the hardware manufacturer it is very accessible. An SDK, Windows App and Windows Twain32 driver, Apple MacOS software.. and a generic UVC interface make it a very attractive learning project.

p.s. I took a look at the MacOS El Capitan [Beta] software as well and it seems to be a perfect feature match, minus only the Save as TIFF feature for the moment. It is not bug free, there are a few bugs, but with patience and experience it is very usable. The "skinnable" user interface Language translation files are also there "in a way", they are not XML files, but plain resource files. I haven't played with them yet.. but if they behave the same I could likewise adapt them for myself.

There are a few reasons for pursuing a Document scanner/Book scanner at the low end of the market. First they have dramatically fallen in price and the software and methods have been time tested and collapsed into a very few steps. Where we used to worry about "eternity" we now generally only worry about the next few years and accept the archival formats will not be "perfect". So archiving a cookbook, school book, or favorite magazine seems much more likely. Dressed up as a stylish desklamp or as more functional furniture than high speed book "ripper" it resonates practicality.

Some might ask why not use a "Point and Shoot" camera, or a moble phone.. but the problem there is one of lighting and consistent repeatability. A camera or mobile phone lighting stand could be tossed into a closet and only pulled out when needed. But since this already looks like a "smart" desklamp.. why not use it as a document scanner as well.

For the cubicle bound office worker this is especially appealing since options for local storage come down to a desk drawer or precious overhead storage for the elite. In which case the Czur scanner as a personal desktop option always ready for use is especially appealing.


LVM, Virtual Volume Management

I just got back from renewing my training experience with Red Hat Linux.

Its always full of new stuff, and this time it included details about RHEL 7.0

We haven't fully adopted RHEL 7.0 yet, but its on the horizon. If RHEL5.0 were Windows XP then RHEL6.0 would be Windows 7 and RHEL7.0 would be Windows 8 or 10. There is so much change.

But one of the things that hasn't changed is the use of LVM.

LVM stands for Logical Volume Mangement, and I had a revelation.. which I have not had before. I know not everyone 'gets it' and some even fear it. But after this 'new point of view' you might like it too.

I was sitting there in class and noticed that LVM parallels what we do with fdisk/gdisk, make file system and so forth. It was like the older partitioning tools were handling the 'Physical Volume Management' or 'PVM' where the 'LVM' tools were handling dicing up the Virtual Volumes that were created from the 'Physical Volumes'.

Its not really quite as simple as that.. but it forced a tunnel vision like focus.. that really LVM was just a 'terribly' poorly named system of tools for performing this 'Virtual Volume Management' function.

LVM truth be told is not its full name, its actually LVM2 because there was an earlier attempt called LVM1 or generically 'EVM' for 'Enterprise Volume Management'.

Enterprise Volume Management was needed because the 'Enterprise' could not afford down time, and needed to speed up the maintenance of replacing drives, extending or shrinking file systems on a live system.

This is critically important in Server class hardware systems, but why is it also useful to Desktop users?

Because the same abstraction allows re-swizeling or reallocating more or less drive space to 'Volumes' which can contain either the entire '/' root file system or any compartmentalized portion of it, like a /home branch.

Its also really nice when drive sizes out strip the ability of BIOS or a particular version of UEFI to access a block on a physical disk and need a shim, or driver provided by the hardware vendor to make it accessible.

The crux of the 'learning curve' however for newbies seems a pathological 'need' by instructors to make it sexy or 'include more stuff'.. usually tacking on things like MDRAID or synthetic software RAID, or more resilient file systems like journaled file systems in the discussion.. distracting and confusing.. and blending the information into this monolithic mess.. that leave a lot of smart people thinking they are inherent or part and parcel of the LVM system.. they are not.. they may depend on LVM to some degree.. but only as much as a file depends on a file system (any file system) for storage blocks.

So what is LVM ?

Put simply, its fdisk for Virtual Volumes.. or fdisk for Volume Groups.

You see to create a Virtual Volume (aka a Volume Group) you first need building blocks, these are called 'Physical Extents' or 'PEs' in Physical disk terms they are 'blocks' of storage. They can be made from carving up whole disks, or carving up MBR partitions of disks, or GPT partitions.. it doesn't matter.. its an abstraction of the 'Physical Blocks on Disk' to virtual 'Physical Extents'.

Once a whole disk or partition has been carved up into PEs they are then 'used' to compose or build a 'Volume Group'. (and you can 'size' these PEs independent of the block sizes on the Physical disks underneath this abstraction, at pgcreate time you can define the PEs 'size' in bytes)

A Volume Group then is like this 'Virtual Hard Disk' which like a thanksgiving turkey, needs to be carved up into smaller Virtual Partitions before you can use them, or mount them in your operating system. Those Virtual Partitions are then called 'Not Logical Partitions' but 'Logical Volumes'.. I can hear you scream.. why.. but why? are they not logically called 'Logical Partitions?' -- well that's because that term is already used down in the cellar of Physical Volume Management.. and we would not want to get confused using that term again. An MBR disk can have four and only four Primary Partitions, after that you cannot create any more.

So planning ahead you can use one of your remaining Primary partition slots to make a special 'Extended' partition.. which is never actually addressed.. except as a pointer to a chain of 'Logical Partitions' underneath... these Logical Partitions (have absolutely nothing to do with Virtual Volume "Logical Partitions").

Sooo.. waay up Topside..on top of Virtual Volumes made from Volume Groups.. we call them 'Logical Volumes'.. ironic.. irritating.. and confusing. (they're [partitions] for cotton pick'n sake..! )

Let me state that again...

A Virtual Volume (which "really" is a Volume Group) is called a Logical Volume.. grrr.

The tools for performing this magic are exceedingly simple.. but hard to remember until you master their names.. and reasons for the choice of their names.. even if that reason is rather obscure and never really discussed.

First Physical Extents (the building blocks) are "made" using the pgcreate tool. (why isn't it called pecreate? I have no idea.. grrr)

Then the Volume Groups (the virtual hard drives or "volumes") are "made" using the vgcreate tool.

Finally the Logical Volumes (the virtual hard drive "partitions") are "made" using the lvcreate tool.

1. pgcreate - "pe create" - makes virtual "leggos" or "virtual disk storage blocks"
2. vgcreate - "vhdd create" - makes "virtual hard drives"
3. lvcreate - "lpart create" - makes "virtual (logical) partitions"

Each tool has a corresponding sister tool called "xx-display" to inspect the results and keep track of the "Virtual environment"

1. pgdisplay - "pe display"
2. vgdisplay - "vhdd display"
3. lvdisplay - "lpart display"

Now once these "virtual volume - (logical) partitions" are created they can be accessed from the /dev or /dev/mapper points just like physical hardware.

And the same tools used for creating a file system can be used on logical volumes to create file systems. Mkfs could then be used to lay down a fresh xfs file system and will be handled by the kernel device driver for xfs just like a physical hardware device file system.

# mkfs -t xfs /dev/vg-group1/lv-volume1

(think of it "like" )

# mkfs -t xfs /dev/vhdd1/lp1


# mkfs -t xfs /dev/vg1/lv1

Then the mount command or the /etc/fstab file can be used to attach the new device and connect it to a mount point on the current file system.

Anything that happens below the "virtual volume" or "volume group" layer.. will be hidden or transparent to the activities of the overlaid pavement of the logical volume (aka the virtual volume 'logical partition' ) and file system.. this is the 'Enterprise quality feature,, which desktop users can also use'

If we need to add more space to a full Logical Volume file system.. we can simply add a hard disk, carve it up into more PEs with the pgcreate command and add those to the volume group, then use an LV tool called "lvextend" to make the partition "bigger" while the file system is being used.. and without backing up the contents and resizing the file system and then restoring the files (a lot of maintenance down time).

Likewise, if we need to "remove" or "replace" a disk (perhaps it failed, is failing or S.M.A.R.T. tells us its expected to fail or some other reason) we can use pvmove ( it stands for 'physical volume Move' why not PEmove ? I have no idea...), to clean out all of the PEs from one disk or partition that is part of a volume group.. without notifying the upper layers, like the LV or file system.. or user.. this "frees up" the physical hard disk or partition and we can take it out of service and replace it. All while the system is running.

The major difference between 'Enterprise' and 'Desktop' is really in the details of whether 'While the system is running' means 'hot' as in 'Live to the world' or 'warm' as in 'Being used but can be rebooted to perform some quick task then back to service'. The game is to minimize system unavailability.

MDRAID or Multidisk RAID (aka software raid) or its likewise drivers can use LVs just like real physical disks or physical partitions to create fast RAID 0 or slow but resilient fail safe RAID 1 drives or anything in between. But they really don't require LVM.

LVM can also do nice things like make Copy on Write or Snapshot images possible.. but those are not fundamental reasons or purposes for LVM to exist.

Including obscure things like ( MDRAID, CoW, journaled file systems et. al. while 'sexy' ) in a newbie introduction simply flys over the important details of LVM and serves to confuse newbies about a very important tool that has become essential in daily life.

The terminology is a quagmire of a historical word swamp and does nothing to make it understandable.