Author Archive: Paul Morgan

Please version a REST API

Martin Fowler says to use versioning as a last resort:

Some people, when confronted with a problem, think “I know, I’ll use versioning.” Now they have 2.1.0 problems.

The problem is that versioning can significantly complicate understanding, testing, and troubleshooting a system. As soon as you have multiple incompatible versions of the same service used by multiple consumers, developers have to maintain bug fixes in all supported versions. If they are maintained in a single codebase, developers risk breaking an old version by adding a new feature to a new version simply because of shared code pathways. If the versions are independently deployed, the operational footprint becomes more complex to monitor and support.

I don’t agree.  I’m a coding dinosaur who’s had to support commercial software releases on old 8″ floppies.  Maybe I am that naive RESTful architect he’s talking about but:

  • With versioning you don’t make modifications to all supported versions for a bug fix.  As he says don’t use a shared code pathway because that’s a bad source control model.  Treat the older code supporting that version as immutable.  Do document bugs in older versions so clients are aware of them.  Don’t modify the code for those versions.  You risk instability from cascading bugs caused by your bug fix.  Those versions are now cut in stone and don’t change.  Keep that code in place until that version of the API is no longer used (a problem for another post) then you can safely remove that version and the code behind it.
  • A bug fix is just another version.  Fixing a bug is installing a newer version of the software.  Some bugs need to be fixed but not in the older versions.  Even if a client is demanding you fix that version you must resist doing that.  Point them to the newer version with the bug fix.  A lot of bugs are benign, low priority or have workarounds so it’s best to just leave them in situ.  If you’re also a client for other micro services  behind your API then the operational footprint isn’t that complex.   If those services are also versioned then it’s just versioning all the way down.
  • With this you never have 2.1 problems using versioning.  Instead you’ll have many many smaller versions – 1, 2, 3, 4 ad infinitum.  They all could have different or the same bugs – some major, some minor.  Resist any pressure to create a 1.1 version when you’re already on version 5.  You replace instead of modify for a micro service (Martin Fowler footnotes this as Replaceable Component Architecture from Dan North).   I also saw this replacable micro service attitude at a recent presentation at the Boston Scala Meetup by Yoni Goldberg at Gilt.  For Gilt they could implement a later version safely in Scala instead of Java.  Let’s just keep the older versions around so we don’t break any clients using an older version.
  • With versioning you have a fail safe if a new version of the API has a serious problem that takes it down.  Just have the clients revert back to the older version and work on a newer version that does work.
  • With versioning your clients are calling an API version then it’s safe for you to install newer versions – this supports continuous deployment of software changes with lower risk.  You decouple any upgrade of the clients from an upgrade of the API.  The Client left hand doesn’t need to know what the API right hand is doing and can find out later.
  • You get to do smoke testing of the API after it’s in production and before clients are using it.  This assumes they call the API with a version.
  • Your clients get to decide when they use the new version.  A lot of customers don’t want to mess with upgrading their systems.  It’s a risk for them and they don’t care about the newer functionality or bug fixes.  They can just leave well enough alone and stick with the older version.

Do It Yourself White Board

Home Depot or Lowes sells large 8′ by 4′ panels of what they call hardboard paneling.  This is very inexpensive at just over $13 a sheet. Oddly enough the surface works just great as a white board.  Compare this to a white board of the same size which is around $300.

If you clean the surface with a regular cleaner like Windex or 409 it will strip the surface and you’ve got a permanent marker board. But if that happens just go buy another replacement panel.

This stuff is also light enough you can hang it on a wall with double sided tape.  Be careful with a painted wall as you can peel the paint off when you pull the panel off the wall.  I’ve also mounted these panels with molly bolts.

When I worked at Online Software they had classrooms with the entire front of the classroom paneled with white board.  I copied that idea but used this less expensive paneling.  We paneled an entire conference room at The Price Club by putting the panels on the wall standing up. We had to cut down some panels to fill the entire wall of the conference room.  Have a pile of markers and dry erasers on the conference room table.  Anyone can stand up, turn around and start diagramming on a floor to ceiling (8′) white board.  Very cool.  

Subsystem or System

I was working at Builder’s Square back in the mid 80s and got tasked with converting the existing payroll process for the company.  We used large 11 x 17 forms for the  employees to fill out which was then sent out to ADP.  These forms had information such as address, bank routing numbers for direct deposit, with-holding for taxes, etc.  They also sent the weekly hours for store employees out to ADP for them to cut checks.

Builder’s Square was a subsidiary of K-Mart and the word had come down that K-Mart’s payroll department could do the work.  They had a custom record layout already created for us to submit all that data on those large forms up to K-Mart in Michigan.

There was also a developer up at K-Mart who was taking the uploaded data and importing it into their payroll system.  We also had a payroll manager up at K-Mart who did the project management and coordination.  There was also a couple of trainers at Builder’s Square who went out to the stores and showed the payroll clerks what needed to be done.

I had to work with developer at K-Mart to include anything on the ADP form into their record format.  Also they had assumed we’d always send everything on every record but I modified the format to fill any field not actually being updated with null values or low values.  No sense in causing an update for a field that hasn’t changed.

I created data entry programs to update the employee data and allow the store payroll clerks to enter the hours.  Any field that was changed got logged into a transaction file to show when it was changed and who changed it.  This file was then extracted at the end of the week to create the change records to K-Mart.  The change records and payroll hours where sent via model to K-Mart every Sunday which took about two hours.

Working with the Avery form vendor rep the two of us modified that large payroll form down to 8.5 by 11.  It was still used but instead of sending the forms off to ADP they now sent them to Builder’s Square corporate payroll.  Modifying that paper form also allowed me to match up the green screen data entry programs to match the form layout. 

We used a dedicated modem to transmit the changes and payroll hours each week to K-Mart.  The transmission took about four hours on a Sunday which was acceptable.  I also had to code the transmission programs to hand shake over SNA to the K-Mart mainframe.  

The next day K-Mart would send down the current employee data file which we would continue to maintain at Builder’s Square.  This kept the two systems in synchronization.  K-Mart had the master copy of the employee data while Builder’s Square had a local copy.  If any problem had occurred where we missed a change or something it would be obvious and we’d have to key the changes in again.

So we got everything working and folks were getting paid right.  Occasionally there would be a problem with the store connection going down.  I had a workaround where the store payroll clerk could drive to a nearby store.  I had used the device description the payroll clerk signed in on to limit them to just that store’s employees.  When a payroll clerk would go to another store I’d do a temporary override to let them maintain one store’s employees from another store.  It wasn’t very pretty but it worked.  With a bit more time I’d have given that override ability to the corporate payroll department and gotten myself out of the loop.

At the end of the project Mike Smith who was the VP if IT at Builder’s Square made a comment to me that ADP did the payroll cheaper that K-Mart.  I got the impression maybe he didn’t want us to succeed with the project.  I replied that it was in house and was probably cheaper for K-Mart.  He’s looking at it from Builder’s Square perspective.  I looked at it more from K-Mart’s point of view.

Was Builder’s Square payroll a subsystem that was needed to be optimized for K-Mart or was it a system that should have been left alone as the conversion to K-Mart wasn’t a cost improvement? 

Version in a Web API

In Web API Design they say on page 13 to “Never release an API without a version and make the version mandatory.”  I don’t agree with this statement.

If I’m a developer I want the freedom to include or exclude a version from the URL.  Stick the version on the end and make it optional.  If I want to always get the latest and greatest from the API then I can leave off the version.  Maybe I’m just doing testing and always want the latest.  It’s possible the API may break but that should be up to me to decide.  Don’t force me to include the version by making it mandatory.

If someone is doing continuous deployment I could have many small changes in the API.  Usually the code supports both the old version and the new version of the API.  Even if I’m forced to include the version in the API the old API version can still break. Putting the version in the API will reduce the chances of breaking the API but it’s not completely avoidable.

A version number is really just an alias for a date/time stamp.  If I’m doing continuous deployment I won’t declare a version until I think enough changes have been released into the API to warrant it.  If you allow a date as well as a version in an API a client can use some intermediate slipstream version. So, using their account examples a versioned URL could be:

  • /account (always the current version)
  • /account/20140524015300 (ISO date format as repeated in RFC3339 but without the required date separator, T & Z)
  • /account/20140524 (simple ISO date)
  • /account/05242014 (USA date format of MMDDYYYY)
  • /account/V1 (with V1 as an alias to something 2014-05-24T00-00-00Z)
  • /account?v=1 (The facebook version as a query parameter is ok too)

It’s also possible to include an Accept-Datetime in the request header (why isn’t this date an ISO or RFC3339 format?).  If there are mutiple versions present I’d say the precedence is URL, query parameter, and header.

You could also include a version in the Accept content type. But you could have a mime type version independent of the resource version.  I’d be cautious about using a version in a mime type.

I’d accept any and all possible versions to make the API more robust.

Comments about versioning I referenced while writing this:

 

 

All Retailers are Different

When I worked at the Price Club we started looking for a replacement merchandising system.  Oddly enough the project was called NMS for New Merchandising System.

One of our requirements was for Average Weekly Sales.  We looked at a software package used by Boscovs which is a regional department store in Pennsylvania.  It appeared to fit the bill for our requirements.  We asked if they had Average Weekly Sales as a check off on our requirements.  The vendor replied yes.  Awesome, check that one off.  Other questions got good answers so we thought we had a good fit.

When we got the software installed we soon found out that we really required Average Weekly Sales by Item by Warehouse/Store. Boscovs, being the regional retailer they were and not too spread out, only had Average Weekly Sales by Item.  Not by location?  Whoops!  We didn’t go deep enough into their data model to discover that issue.  We might have also assumed that everyone had Average Weekly Sales by location so didn’t bother to ask if it was by location.

The idiom “the devil is in the details” comes to mind.

So my lessons learned from this was:

Look at the underlying data model for a software package to match up your requirements.  They had Average Weekly Sales but it wasn’t an attribute on the right entity in the data model.

Be careful about any assumptions (even hidden) you might be making.

 

Google Cloud Print

Google Cloud Print

This is a really great service.  I’ve been able to give the whole family access to our color printer attached to my desktop computer at home.  Previously they couldn’t print using WiFi because I’ve segregated the WiFi computers at the router.  Now they can print anywhere and anytime.  They don’t even have to be in WiFi range. We can print from our cell phones, laptops, and notepads.  Pretty dang slick!

Why can’t collections also be a composite?

Collections like Array, List, etc which are groupings of objects.

Composites are used in hierarchies to give the same behavior to the composites class as to the leaf/node class. The example from the GoF has both the leaf and composite classes using a common base class or interface.

Many objects in systems I’ve worked with have objects grouped in hierarchies. For a Merchandise hierarchy Items are in Categories, Categories are in Departments, Departments are in Companies. For a Location hierarchy Stores/Warehouses are in Regions, Regions are in Divisions, Divisions are in Companies. For a Calendar hierarchy Days are in Weeks. Weeks are in Periods, Periods are in Years. Categories, Departments, Divisions, Weeks, Years and Companies are all composites.

I can also make an array of Items or Stores or Days. Using that array I can iterate through every objects in the array and call a method/function on each object. But why iterate through the array of objects? Why can’t I just treat the array as a composite which has the same methods/functions as the objects in the array. Instead of iterating through the array I just call the method/function on the array directly and let the array do the heavy lifting of calling that method/function on each object in the array.

I also want to use the array as a duck type of the objects in the array. Assume a function expects an object of type Item. Add some Item objects to a new Array. As an Array of Items is also an Item that Array can also be passed to the function. The Array of Items is dynamically typed. The Array of Items isn’t a composite of Item until you place an Item into the Array at run time.

If a method/function returns a value there could be some way to handle the values returned by every object in the Array. You could just ignore the returned values and return nothing from the Array method/function. Or you could assign some summary operation similar to the summary operations in SQL – Minimum, Maximum, Summation or Average. Or you could assign some arbitrary summary function that accepts a series of returned values and returns a single value.

Is there any language out there that supports this dynamic typing of an Array to the class in the array?

About the name

Nimble Union was originally a guild name for World of Warcraft.  But I’ve been working on an Agile team and like the name as a working name for my development.  Nimble as a synonym for Agile.  Union as a metaphor for a self organizing team.  Right now it’s just me so I’m being a bit pretentious calling it a union but I have dreams.