testingReflections.com - design & development

Software Performance Engineering

Tue, 11 Nov 2008 22:34:31 -0600

Trying to make some suggestions for Building Responsive and Scalable Applications panel I contemplated for a while on this topic.

For a long time we have Software Performance Engineering addressing these issues. As it described in the books and papers of Dr. Connie Smith and Dr. Lloyd Williams, it looks rather as a methodology for me.

The short and the long of IT: two videotaped presentations

Tue, 14 Oct 2008 17:28:09 -0500

Since the middle of the year, I’ve presented and facilitated about 12 hours of sessions at 2 traditional and 3 open space conferences, plus a Googleplex visit. Two presentations are now on video, both filmed on the other side of the world from my usual location of Melbourne, Australia.

A lightning talk (at the functional tools workshop held before Agile 08 in Toronto, Canada) called Shades of Green discusses how the “green” passing tests of functional automation may not be as green as they seem. Note the static pose to stay within camera range, compensated for by the wildly waving arm. And yes, the audience was not limited to a leg and a foot.

An order of magnitude longer at around 50 minutes, a Google Tech Talk (filmed at the San Francisco Googleplex) called 80:20 Rules! Building Software Smarter looks at formal and informal ways to get significant improvements in creating software, including various puzzles and questions for viewers. I had looked at some tech talks by other people I know, and they had been watched around 1000 times over a year or so. It looks like I may hit that mark only a few days after the video was posted which is great. I hope my talk inspires people to build their software smarter. Can I turn “shades of green” into a similar talk? Probably not!

ISubmitBlogPosts - a nice twist on Apps Hungarian Notation for Interfaces...

Tue, 18 Nov 2008 13:26:44 -0600

I've been pairing with Andy Palmer over this last week. I have to say it's been a lot of fun... and I think we've learned a lot from the experience. One of the things I learned from Andy this week was an innovative use of Hungarian notation for interfaces... Andy told me about Udi Dahan's presentation on intentions and interfaces (PDF)

Pitfalls of the "Waterfall" Approach to Performance Testing

Mon, 24 Mar 2008 16:23:17 -0500

Looks like the pre-production validation approach to performance testing becomes typical for large corporations (if there is any at all):

-get the system ready
-develop all scripts requested (sometimes offshore)
-run them all together
-compare with the requirements provided
-allow some percentage of errors according to the requirements
-involve the development team if requirements were missed

The P28 virtual fence: Borderline success or virtual vaporware?

Wed, 05 Mar 2008 04:06:01 -0600

I’ve already noted some of the issues around the virtual fence here. In May 2006, George W. Bush called it “the most technologically advanced border-security initiative in American history”. In a flash of originality, the 28 mile virtual fence in Arizona was called Project 28, or P28 in Chertoff-speak. Micheal Chertoff has now accepted the fence system, on the basis that “all of the defects” in the prototype project were either “cured” or “immaterial.” He also said “In some form or fashion, technology is going to be virtually every place on the border, but it’s not necessarily going to be in the configuration of P28,” The chair of the House Committee on Homeland Security, Representative Bennie Thompson, sees it differently, “The poorly structured contract that prevented the line Border Patrol agents from pointing out obvious flaws and caused an overreliance on contractors has resulted in a system that has been described as providing ‘marginal’ functionality at best,” I wonder if he was tempted to say the Border fence had borderline functionality. [grin]

The technology stack on top of each tower looks impressive . The project is anything but impressive , with many of the issues typical in the IT industry: badly underestimated effort, not involving users, and not enough testing, While there was evidently a push for reuse of existing systems and components, it caused many difficulties: trying to base the system around a law enforcement dispatch system (!) that couldn't scale, trying to use off-the-shelf components that weren't designed to integrate, and a lack of standards for the sensors. The builder of the system, Boeing, has only taken 3/4s of the $20 million fee, granted a $2 million credit, and apparently paid $40 million to get this far on a project that will now finish in 2011, not the end of this year (which was already a year and a half past the original schedule). That will only be phase one!

CMG Opens Its Content to the Public

Tue, 12 Feb 2008 13:30:21 -0600

The Computer Measurement Group (CMG) is making its conference proceedings from 1997 through 2005 available to the public. I believe that CMG holds the best practical conference in performance analysis, capacity planning, and related areas. In addition to the areas listed below, I'd definitely add performance testing. Here is the official mail CMG sent:

Education

Sat, 02 Feb 2008 23:08:05 -0600

With great interest read Who Killed the Software Engineer? (Hint: It Happened in College) by James Maguire as well as the original article Computer Science Education: Where Are the Software Engineers of Tomorrow? by Dr. Robert Dewar and Dr. Edmond Schonberg.

Performance Requirements

Sun, 06 Jan 2008 17:08:27 -0600

My performance requirements paper was published in the January issue of Software Test & Performance (pp.18-24).

It was simple: I just sent a draft – and now I am reading it printed. With a new name - or even two: it is referred as You Can Gauge Performance Without Requirements in one place and Gauging Performance in The Absence of Measures in another. Not to mention other minor improvement.

From the Mailbox: Software Development: Art or Science?

Thu, 18 Oct 2007 14:47:12 -0500

Here’s a question that I didn’t realize I had much to say about until I read my own response.

The Question:
Software Development: Is it an art or a science? An age old question I know, but what do you think and why?

My Response:

I refer to new software development as a scientific art. I've seen some maintenance work, platform porting, etc. that has been almost entirely mechanical -- I'm not sure what that counts as, but I certainly didn't witness anything "artistic".

What Skills Performance Testers Need and How to Get Them?

Mon, 15 Oct 2007 09:31:39 -0500

From time to time I see questions on different forums asking what skills are necessary for performance testers. There were pretty interesting discussions. Looks like most experts agree that performance testing requires more skills than just knowledge about how to create a script for a particular load testing tool. While it is still possible to imagine a performance tester in a large corporation with deep specialization who only creates scripts and mechanically runs them while other performance experts monitor the system and analyze results, I don't see many perspectives neither for this person, nor for the approach. Systems become so complicated now that the sum of specialized expert views doesn't give the whole performance picture.

Guerilla Approach

Thu, 04 Oct 2007 14:06:44 -0500

Mark Friedman reviewed Neil J. Gunther's new book "Guerilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services"
in MeasureIT (free registration may be needed - I hate registrations, but this one is worth efforts if you care about performance: it is a very good newsletter).

"Shot in the Dark" anti-pattern

Fri, 01 Sep 2006 21:00:50 -0500

Kirk proposed the "Shot in the Dark" anti-pattern. By my opinion, a must reading for all working with performance issues.

Predicting the Path of the Storm

Tue, 27 Jun 2006 12:11:11 -0500

Recently, three unrelated events have given birth to a new idea in my head, and I wanted to share it with you. So, to properly set up the background, the three unrelated events:

At the recent CAST, James Bach presented a keynote, “Against Certification”. During that presentation, one of the things he did was to review some of the “Body of Knowledge” documentation upon which such certifications were based. As he reviewed this documentation, one of the things I was struck by was an underlying pattern of motive driving much of the documentation. That motive was the ability to predict, manage, and control the SDLC. Much of this “Body of Knowledge” for test certification was obviously written by managers, not testers.

Python - log parsing, statistical analysis, and performance graphs

Sun, 04 Jun 2006 16:07:26 -0500

Python - log parsing, statistical analysis, and performance graphs:

Over the past few years, Python has become my favorite language to program in. I do tools development and work regularly on a variety of platforms, so I try to stay versatile with respect to languages. I've written lots of code in a variety of languages over the years (C, C++, Scheme, Pascal, Java, C#, Perl, Python, etc.). For a long time, Perl and Java were my goto languages. Any quick script or heavy text slinging; I would reach for Perl. Any larger project that needs an organized class structure; I would reach for Java. Then a few years back I started banging around Python and learned what a fantastic programming language it can be for _anything_.

The past few days I have been working on a tool for analyzing performance data that an application logs during load tests.

I needed a working version quickly that I can show as a proof of concept. Part of the requirements is that the tools must integrate well with a .NET environment, and be maintainable and extendable by people in the .NET shop. Many people claim Python is a great prototyping language. It has clean syntax and structure and is great for quickly building class libraries. I started to think I would create something quick in Python and then maybe later port it to C#. Well it turned out so nice and so easy to work with, that I can't imagine using anything but Python for it now. (IronPython perhaps?)

so I ended up creating a generic framework in Python and exposed a scriptable API.

It can:
- parse MS Event Logs
- slice the data up into a time-series
- run some statistical calculations on the time series
- output graphs (to gif/png images for web display, or to a GUI with more powerful viewing)

It is built with:
- Python
- Matplotlib
- MS Log Parser 2.2

I had to do a lot of work with crunching data sequences and slicing up time series data.
Python's dynamic typing and simple data structures made it very flexible to handle all the data processing with a minimal amount of code. The most useful thing was Python's List Comprehension features. List Comprehensions are very powerful constructs for list processing that allow you to do some heavy lifting with numeric sequence processing in a very concise way.

MS Log Parser 2.2
Log Parser is a tool from Microsoft that lets you query log files with an SQL dialect. I built a wrapper around this with Python's popen methods.

Matplotlib
Matplotlib is a 2D plotting library written in Python. I created a graphing API for my framework that uses this. It was simple to create and the graphs look great.

corestats.py
One of the classes I wrote was for doing simple statistical calculations. You can grab a copy here if you are interested: http://www.goldb.org/corestats.html

-Corey Goldberg
www.goldb.org

Performance vs. Testability and semi-virtual method calls

Mon, 25 Jul 2005 00:41:26 -0500

I won't try to sit here and say that the issue of performance vs. testability can be summed up in one blog post (or a full article for that matter). However, it's an important one and during my consulting gigs I've come across some interesting questions and answers regarding this area. Here are some of my thoughts on this. In languages like C++ developing real-time performing applications may also mean having coding-call standards such as "do not use virtual functions" in the feat that a virtual call repeated enough times may slow the system substantially.