doug — off the record

just a place to share some thoughts


On the Shoulders of Giants

I had a nostalgic moment this morning, and yet at the same time, it made me feel so humble.  It was one of those moments that those of us who are technical longtimers would get – and shorttimers might not even recognize.

When I went to bed last night, I was going through the database of session proposals for the #BIT14 conference.  The whole process is just taking so long – there are so many good proposals and only so many timeslots.  We collected the proposals on the web and I had downloaded them to my computer and was going through the list for what seems to be the 100th time.  The proposals are in .xlsx format and I had them open in LibreOffice as I read.  It was late, and as I typically do, I just shut the lid on my computer sending it to sleep before I joined it.

When I woke, it was time to pick up where I left off.

The only difference was that there was an update to the LibreOffice program.  

Oh boy!  At 181MB, this is going to take a while.  My “high speed” high speeds have been well documented.  I might as well get it started.

There were lots of other things to be done so I initiate the download and moved on.  I didn’t time it but there came a time when the download was done and I installed the new software.

It was there that I had my first aha! moment of the day.

The 181MB download is actually compressed and, in reality, the application is ~580MB.  I think we’ve come to expect that things are going to be compressed for our benefit.  But, in the beginning, it wasn’t always so.  You had to work very hard at compressing and decompressing files.  After all, you were squirting them through a 1200 baud modem and every byte  counted.

Compression is now everywhere.  But there was a time when you really had to work at it.  For me, it was a product called PKZIP.  It was shareware but you just had to throw a few bucks their way to get the functionality that you needed.  You could archive entire directories, compress them, send the resulting file to someone else who would reverse the process and have the exact same installation that you did.  It was an electronic version of magic.  You could encrypt the archive with a key, add a comment to the archive, and even fix some broken archives.  PKWare is still around today and their product line has evolved to reflect our current realities.  

Check out archive file formats.  Do you know them all?  Compression is such a fascinating concept.  I can recall working with Run Length Encoding at university.  We were making the magic!

Compression isn’t just for shipping archives.  It now does disk images, makes graphic files smaller, allows for efficient distribution of movies and television shows.  We’ve come to expect that our files are smaller and efficiently transferred from one place to another.  We’ve even become willing to accept compromises in quality for speed.  We’ve got used to utilities that optimize things for us automatically.  Can you remember working with Photoshop to get the best compromise of quality and size to make your website just speed along to any visitor?  Is it the same in today’s drag and drop world?

It’s worth pausing before the next download to appreciate all that these giant thinkers have offered to us.  Where would we be without their vision of makings things smaller and faster?  It’s hard to imagine a world and a digital routine without it.

I’d still be downloading that update.



2 responses to “On the Shoulders of Giants”

  1. I recall using a variety of archival/compression formats in the early 90s. We used ZIP and ARJ to try to save space for savegame files so they would fit on our floppy disks. ARJ was a great format for us, consistently compressing better than ZIP. I once wrote a C program to write run-length encoded TARGA files; that was interesting. I currently use 7Zip and PeaZip Portable in Windows. Some comics and graphic novels I buy are distributed in CBR (Comic Book RAR) or CBZ (Comic Book ZIP) formats, which are just renamed from RAR and ZIP. SMART Notebook (.notebook) files are similarly just renamed ZIP files.

    I didn’t really understand how ZIP files could be so heavily compressed, even after learning RLE, until I was in university. It wasn’t a class; I was hanging out with some CS majors I worked with and someone (possibly I) mentioned not understanding LZW.

    30 minutes and a worked example (“How now brown cow?” and so on) showed me/us the exceptionally clever approach the algorithm takes. It’s humbling to see what people are able to imagine, test and prove.

    Efficiency of space is a big concern again. Mobile devices and cellular connections, with their “slow” processors, limited batteries and inconsistent network connections, have reminded developers that it matters how you code things. You can’t expect a user’s device to just churn through what you don’t want to bother doing cleanly.

    For graphics there is an ongoing discussion about the place of SVG and other vector formats. They are a set of instructions about how to draw an image, rather than a list of pixels. And they are “scaleable” (the S in SVG), but dramatic changes in size are bad from a design perspective. If it interests you, check out some of the debate online. There are some very good reasons to keep raster images around, even if vector images are scaleable and have smaller footprints.

    Well, that was a tangent. Thanks Doug – I like a good tangent once in a while. It reminds me of how interconnected concepts are in technology!

    Liked by 1 person

  2. Thanks for sharing, Brandon. I would say that was more of a blog post than a tangent! I’ve typically thought of vector images as the type I would use when working with large posters and sending images to printers. I still find it interesting, when time is on my side, to save in PNG or JPG formats and compare sizes. I like your take on portable. You’re right; that’s a big area that needs to be considered in this discussion.

    Like

Please share your thoughts here. I’d enjoy reading them.

This site uses Akismet to reduce spam. Learn how your comment data is processed.