Progress Bars

Some background information

The progress bar.

Not just the name of a sadly defunct bar in London or a sign depicting the prevention of egress, progress bars serve a useful purpose to inform the user just how long they’re going to have to sit around with a thumb up their arse whilst they perform their highly skilled non-DMCA-based white-collar workloads.

As you may be aware, Microsoft will be dropping support for several classes of progress bar in 2014, for reasons of security to the Microsoft revenue base.

A lot of tasks don’t have an easily-defined halting condition, so you get slightly different progress bars that move a little to the right, and then a little to the left, or look like barberpoles if you use one of those computers that they look like barberpoles on.

These days of course, it’s more modern to expose your end-users to “throbbers” instead, which give a pleasing sense of going round in circles, and are probably a good indication of what limbo is going to end up being like, if you’re religiously inclined.

They appear in many guises, such as

Some foreground information

So anyway, I’m not one to stand in the way of progress (ha), and now that I think you might know what a progress bar is, here’s something to make your scp task generate some javascript which you can dump into a browser to make a little rectangle get larger as the copy process takes place.

People generally expect to see this sort of thing during long periods of what appears to be inactivity, so you might as well give it to them so as not to surprise them, which apparently is a usability nightmare, unless you’re trying to cram the next great evolution in gesture-based interfaces down people’s throats.

The program itself is called scpwrap, and it’s vaguely similar to that previous post about piping stdout/stderr streams in VBA, since that’s what it does, but this implementation is in C, which is much more self-respecting. You could probably drop it into a bash script without people thinking they should wrap it in a chef recipe or a puppet whatever-they-call-their-module-components [1].

It’s slightly more complicated than it should be because scp won’t normally display progress information if it’s not running in a terminal, hence the forkpty pseudo-terminal malarky on line 319. The code is based on (and is backwards-compatible with) the scp-to-zenity wrapper written by Clayton Shepard, but also handles non-standard stdout/stderr streams, has customisable output templates, and deals with error conditions in a relatively sane way.

Here’s the sourcecode for the scpwrap program, and the groff source for the manpage, which took almost as long to write up.

There’s also a link to the randomnoun apt repository at the bottom of this post, if you want to install it automatically.

[rn-sourcecode lang=”c” src=”scpwrap/scpwrap.c” errors=”true”]

[rn-sourcecode lang=”text” src=”scpwrap/scpwrap.groff” errors=”true”]

And here’s the manpage itself:

SCPWRAP(1)                       User Commands                      SCPWRAP(1)



NAME
       scpwrap - a wrapper for scp to convert its output into machine-readable
       format

SYNOPSIS
       scpwrap [--js] [--stdoutTemplate template ] [--stderrTemplate template
       ] [--startTemplate template ] [--progressTemplate template ]
       [--endTemplate template ] -- scp-options ...

DESCRIPTION
       scpwrap wraps the scp(1) command such that the output generated by that
       command  is  converted  into  a  format  more easily processed by other
       commands.

       scp(1) progress information is captured and converted into  a  machine-
       readable format for use in more sophisticated user interfaces (which in
       turn would require even more  effort  to  turn  back  into  a  machine-
       readable format). And so the circle of life continues.

       The  built-in  text  and  javascript  output templates are suitable for
       zenity(1),  or  some  as-yet-to-be-documented   javascript   processor,
       respectively.   See  the  EXAMPLES section to see the general syntax of
       these defaults.

OPTIONS
       --js   Use the default javascript templates  rather  than  the  default
              text templates (see the DEFAULTS section below).

              This  switch  also  enables  javascript-output escaping (see the
              JAVASCRIPT OUTPUT ESCAPES section below).

       --stdoutTemplate template
              Use the supplied template to generate machine-readable text when
              unrecognised output is detected on stdout. The placeholder %s in
              the template will be replaced by the text received on stdout.

       --stderrTemplate template
              Use the supplied template to generate machine-readable text when
              any output is detected on stderr (e.g. to display text contained
              in the remote system's sshd (8) Banner file). The placeholder %s
              in the template will be replaced by the text received on stderr.

       --startTemplate template
              Use  the  supplied template to generate machine-readable text as
              scp progress begins.

       --progressTemplate template
              Use the supplied template to generate machine-readable  text  as
              scp  progresses.   The  placeholders  %f,  %p, %t, %s and %e are
              available (see the PLACEHOLDERS section below)

       --endTemplate template
              Use the supplied template when scp completes. The %c placeholder
              is available.

       scp-options
              these  command-line  options  are  passed directly to scp(1) and
              should conform to the expected syntax of that command.

PLACEHOLDERS
       Placeholders can (and should) be embedded in  template  strings,  which
       will  be  replaced  by  the appropriate value when the stdout/stderr of
       scp(1) is parsed by this utility. Most placeholders are only  available
       in specific template types.

       Recognised placeholders are:

       %s   Free-form stdout or stderr text

       %f   The name of the file being copied (e.g. test.txt)

       %p   The percentage progress of the current file being copied (e.g. 13)

       %t   The transfer size of the current file being copied (e.g. 2112KB)

       %s   The transfer speed of the current file being copied (e.g. 2.1MB/s)

       %e   The ETA of the current file being copied (e.g. --:-- or 05:23)

       %c   The exit code of the scp(1) process (e.g. 0), or a negative number
            if the process was terminated by a signal (the absolute  value  of
            this number corresponds to the signal number received).

PLACEHOLDER ESCAPES
       The  following  escape  sequences are recognised in placeholder strings
       supplied on the command-line:

       \n   Newline

       \r   Carriage return

       \t   Tab character

       \\   Backslash

JAVASCRIPT OUTPUT ESCAPES
       If the text generated by  scp  contains  characters  that  would  cause
       javascript  parsing  errors,  and  the  --js  command-line option is in
       effect, then the strings substituted into the %s  placeholder  will  be
       javascript-escaped  (e.g.  newlines  will be replaced by \n, characters
       above ASCII codepoint 127 will be replaced by \u0000-style sequences.)

DEFAULTS
       The following defaults  are  used,  in  the  absence  of  any  template
       overrides:

   Text output (without --js parameter)
       Note  that  many  templates  are  simply  ""  (i.e.  the  empty string,
       therefore no output is generated)

       --stdoutTemplate    ""

       --stderrTemplate    ""

       --startTemplate     ""

       --progressTemplate  "%p\n"

       --endTemplate       ""

   Javascript output (with --js parameter)
       The default javascript output relies on a script-visible  'ui'  object,
       as shown below.

       --stdoutTemplate    "ui.addOutput(\"%s\");\n"

       --stderrTemplate    "ui.addOutputError(\"%s\");\n";

       --startTemplate     "var sp = ui.startScpProgress();\n";

       --progressTemplate
                           "sp.setProgress(\"%f\", %p, \"%t\", \"%s\", \"%e\");\n";

       --endTemplate       "ui.stopScpProgress(%c);\n";

EXIT STATUS
       The  scpwrap  command  will  return the same exit code as the child scp
       process; i.e. it exits 0 on success, and >0 if an error occurs.

       If a signal interrupts processing of the child  process,  then  scpwrap
       terminates  with  an  exit  status  of  1.  The  signal  number  can be
       determined using the %c placeholder to the --endTemplate template.

EXAMPLES
   Example 1 (text output)
       The command

              scpwrap -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz

       might produce output something similar to the following:

              0
              45
              47
              49

                ... 20 lines omitted ...

              98
              100

   Example 2 (javascript output)
       The command

              scpwrap --js -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz \

       might produce output something similar to the following:

              ui.addOutputError("NOTICE TO USERS\n");
              ui.addOutputError("\n");
              ui.addOutputError("This service is for authorised clients only.\n");
              ui.addOutputError("\n");
              ui.addOutputError("This computer system is the private property of its owner, whether\n");
              ui.addOutputError("individual, corporate or government.  It is for authorized use only.\n");
              ui.addOutputError("Users (authorised or unauthorised) have no explicit or implicit\n");
              ui.addOutputError("expectation of privacy.\n");
              ui.addOutputError("\n");
              ui.addOutputError("It is a criminal offence to:\n");
              ui.addOutputError("  i. Obtain access to data without authority\n");
              ui.addOutputError("       (Penalty 2 years imprisonment)\n");
              ui.addOutputError("  ii Damage, delete, alter or insert data without authority\n");
              ui.addOutputError("       (Penalty 10 years imprisonment)\n");
              ui.addOutputError("\n");
              ui.addOutputError("For more information, see http://www.randomnoun.com/login-banner.html\n");
              var sp = ui.startScpProgress();
              sp.setProgress("somefile.tar.gz", 0, "0", "0.0KB/s", "--:--");
              sp.setProgress("somefile.tar.gz", 45, "2112KB", "2.1MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 47, "2208KB", "1.9MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 49, "2320KB", "1.7MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 52, "2448KB", "1.5MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 54, "2576KB", "1.4MB/s", "00:01");

                ... 20 lines omitted ...

              sp.setProgress("somefile.tar.gz", 98, "4624KB", "266.7KB/s", "00:00");
              sp.setProgress("somefile.tar.gz", 100, "4693KB", "187.7KB/s", "00:25");
              ui.stopScpProgress(0);


   Example 3 (custom javascript output)
       The command

              scpwrap --js --stderrTemplate '' --stdoutTemplate '' \
                --startTemplate '' --endTemplate '' \
                --progressTemplate 'setProgress(%p);\n'
                -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz

       might produce output something similar to the following:

              setProgress(0);
              setProgress(45);
              setProgress(47);

                 ... 20 lines omitted ...

              setProgress(98);
              setProgress(100);

BUGS
       It might be preferable to get the default  strings  from  something  in
       /etc

       The whole thing's a bit pointless

AUTHOR
       Greg Knox 

LICENCE
       (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
       BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)

SEE ALSO
       vmaint(1), https://www.randomnoun.com/wp/2013/10/31/progress-bars/



vmaint                           OCTOBER 2013                       SCPWRAP(1)

If you want to install it in binary form, I've created an apt repository for a couple of ubuntu distribution/architecture combinations (maverick/amd64 and precise/i386), which you can add to your /etc/apt/sources.list file automatically using these commands:

$ sudo add-apt-repository "deb http://packages.randomnoun.com/ubuntu $(lsb_release -sc) main"
$ gpg --keyserver  hkp://pool.sks-keyservers.net --recv-key A21A1486 && gpg --export --armor A21A1486 | sudo apt-key add -
gpg: requesting key A21A1486 from hkp server pool.sks-keyservers.net
gpg: key A21A1486: "Greg Knox " not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
OK
$ sudo apt-get update
Get:1 http://packages.randomnoun.com maverick Release.gpg [198B]
Get:2 http://packages.randomnoun.com maverick Release [2,242B]
Ign http://packages.randomnoun.com maverick/main Sources
Ign http://packages.randomnoun.com maverick/main amd64 Packages
...
$ sudo apt-get install scpwrap
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  scpwrap
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.0kB of archives.
After this operation, 69.6kB of additional disk space will be used.
Get:1 http://packages.randomnoun.com/ubuntu/ maverick/main scpwrap amd64 1.0-1 [12.0kB]
Fetched 12.0kB in 0s (113kB/s)
Selecting previously deselected package scpwrap.
(Reading database ... 59957 files and directories currently installed.)
Unpacking scpwrap (from .../scpwrap_1.0-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up scpwrap (1.0-1) ...

If you don't have add-apt-repository installed in your OS, then try sudo apt-get install python-software-properties first (or you could manually edit /etc/apt/sources.list instead).

Update 4/11/2013: Added instructions for the packages.randomnoun.com apt repository; added BSD licence to source code