Invariant Properties

  • rss
  • Home

Lessons from BIBIFI

Bear Giles | June 24, 2015

I haven’t posted in a while since I’ve been very busy. Coursera classes on computer security, cloud computing and geospatial technologies, corporate initiatives, even an Amazon Web Services certification. It adds up. The capstone of the computer security specialization was a Build-It, Break-It, Fix-It. It was a 6-week effort for working professionals with a wide variety of skills and experience. It was still pretty stressful albeit for different reasons.

It was not my finest hour. But that’s okay since falling flat on your face (well, it wasn’t THAT bad) is when you learn the most. I didn’t learn anything new but before I had only read it in a book. This time I lived it – or more precisely saw in retrospect just how much of a difference it would have made.

(Note: this are general management issues. I’ll probably discuss the security aspects later.)

A clear specification is still critical

Many of us have gotten sloppy since the agile methodology was introduced. We’re used to working closely with product owners and business analysts and the time spent to nail down specifications that change at the end of the sprint is considered wasted.

There’s a lot of benefits to this approach but the BIBIFI round reminded me that we still need the ability to write clear specifications and quickly identify ambiguities in the rest.

In the competition the spec was ambiguous in spots but the problems weren’t fully recognized until long after development had started. It was no longer possible to make changes since there were approx. 60 teams working during their limited free time.

An executable specification is a huge improvement

Many of us are familiar with test-driven development (TDD). In brief the tests are written first and the code is implemented only far enough for the tests to compile but fail. Development only proceeds once the tests have been written and stops once all tests pass.

Tests are still expected to pass before code is checked in, or at most by the end of each sprint.

An executable specification is similar. Acceptance tests are written against the user interface – these tests can use SoapUI, selenium, etc. Internal libraries may only need to pass unit and functional tests. The system is initially only implemented far enough for the tests to compile and run. Development stops once all tests pass.

The key difference is that some tests will remain red until the end. This now looks very odd to many of us, or at least it should. It’s important to remember that it serves a different purpose. This shows how much you have left to do, not whether you have introduced defects with your most recent changes. Some people believe these tests shouldn’t even be visible to developers. We need to see them during sprint planning but in their view they’re nothing but a distraction during development.

In the competition a submission was required to pass several acceptance tests before it would be accepted for the ‘break-it’ phase but the acceptance tests were far from complete. Someone who should have known better (ahem) relied on them instead of writing his own test suite based on the specification. That was a mistake since the acceptance tests were far from complete and key functionality was overlooked.

(One of my recommendations for future rounds is that the acceptance tests be beefed up and then deeming anything that passes as correctly implemented. The uncertainties in our specification and the scoring resulted in many teams focusing on ‘bug hunts’ (with guaranteed points) instead of the much harder and riskier effort to examine code for security flaws.)

Product owner specifications are best

Behavior-driven development (BDD) is superficially similar to test-driven development. One big difference is that it puts an emphasis on writing specifications in terms that are understood by the product owner. The onus is on the developer to work with this format, ideally in an automated fashion.

In the competition a lot of information was provided in JSON files. I understand JSON and these were simple files so I thought I could just view the raw file. In retrospect I often lost the big picture in the minutia.

I became much more productive after I spent a little time writing a tool that converted between the JSON files and Excel® files and could run the tests directly. It was too late to make much of a difference.

What is the best format for specifications?

Easy – the one you will use.

A bit more seriously it needs to be structured but easily used by product owners and non-technical business analysts. In many cases that means Excel® files. They’re flexible without being completely free form (like Word® documents), familiar to everyone, and easily tracked in Sharepoint® and the other tools so beloved by the suits.

Excel® can read and write CSV files – easily parsed – but for various reasons I lean towards using the native xlsx format and using a library like Apache POI to read and write the files. It requires a little more effort on our side but it will be much easier for the PO/BA to maintain. Note: LibreOffice can also read and write these files and I routinely use it on Linux in preference to Excel® on Windows®.

The implementation is fairly straightforward – each sheet is associated with a test class and contains one set of test data. The class knows how to map each column to expected inputs and outputs. A basic system simply reads the Excel® file, a more advanced system can produce an Excel report as well.

The TestNG framework makes it easy to nest test classes so the workbook is opened once, each of the tests is run, and then the workbook is closed and any report is written and published.

Final note

I don’t know if Apache, LibreOffice, TestNG, and others want the ® annotation. I’m pretty sure Microsoft® does!

Categories
security
Comments rss
Comments rss
Trackback
Trackback

« Tips for Importing Data Auto-encrypting Serializable Classes »

Leave a Reply

Click here to cancel reply.

You must be logged in to post a comment.

Archives

  • May 2020 (1)
  • March 2019 (1)
  • August 2018 (1)
  • May 2018 (1)
  • February 2018 (1)
  • November 2017 (4)
  • January 2017 (3)
  • June 2016 (1)
  • May 2016 (1)
  • April 2016 (2)
  • March 2016 (1)
  • February 2016 (3)
  • January 2016 (6)
  • December 2015 (2)
  • November 2015 (3)
  • October 2015 (2)
  • August 2015 (4)
  • July 2015 (2)
  • June 2015 (2)
  • January 2015 (1)
  • December 2014 (6)
  • October 2014 (1)
  • September 2014 (2)
  • August 2014 (1)
  • July 2014 (1)
  • June 2014 (2)
  • May 2014 (2)
  • April 2014 (1)
  • March 2014 (1)
  • February 2014 (3)
  • January 2014 (6)
  • December 2013 (13)
  • November 2013 (6)
  • October 2013 (3)
  • September 2013 (2)
  • August 2013 (5)
  • June 2013 (1)
  • May 2013 (2)
  • March 2013 (1)
  • November 2012 (1)
  • October 2012 (3)
  • September 2012 (2)
  • May 2012 (6)
  • January 2012 (2)
  • December 2011 (12)
  • July 2011 (1)
  • June 2011 (2)
  • May 2011 (5)
  • April 2011 (6)
  • March 2011 (4)
  • February 2011 (3)
  • October 2010 (6)
  • September 2010 (8)

Recent Posts

  • 8-bit Breadboard Computer: Good Encapsulation!
  • Where are all the posts?
  • Better Ad Blocking Through Pi-Hole and Local Caching
  • The difference between APIs and SPIs
  • Hadoop: User Impersonation with Kerberos Authentication

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Pages

  • About Me
  • Notebook: Common XML Tasks
  • Notebook: Database/Webapp Security
  • Notebook: Development Tips

Syndication

Java Code Geeks

Know Your Rights

Support Bloggers' Rights
Demand Your dotRIGHTS

Security

  • Dark Reading
  • Krebs On Security Krebs On Security
  • Naked Security Naked Security
  • Schneier on Security Schneier on Security
  • TaoSecurity TaoSecurity

Politics

  • ACLU ACLU
  • EFF EFF

News

  • Ars technica Ars technica
  • Kevin Drum at Mother Jones Kevin Drum at Mother Jones
  • Raw Story Raw Story
  • Tech Dirt Tech Dirt
  • Vice Vice

Spam Blocked

53,793 spam blocked by Akismet
rss Comments rss valid xhtml 1.1 design by jide powered by Wordpress get firefox