rawdog

Thursday, 19 November

08:36

Time, Time, Time, See What You've Done To Me [Testy Testy]

Today I found and argued about and isolated many bugs on one topic alone. Time.

Testing tools used:
1 calculator (free with Windows AND Mac operating systems)
System Event Viewer and Advanced time control panel for Windows.

Problems:
Times listed in steps did not add up to the "overall duration" ever. We are missing some time.
Client time is used for some calculations causing things to happen in negative time for some time zones.
The "time" server lied making things happen in the far future for negative time.
Some time is in GST and some is not which is problematic.

Good Things:
Time is always tracked in seconds (yay), so the displays in various ways all work.
Date last modified is shown and consistent.

Questions: What else should I ask myself and find out about time overall? Are there other tools which can help me test time I should look into? What interesting time related issues have you found?

In short, I missed my bus, am behind on other tests, and still I am interested to see if this software can stand the test of time. Yes, pun intended.

Context Driven [Testy Testy]

I am swamped today at work and was up fretting pointlessly about my future last night rather than being my usual cheery self, but something struck me strongly today. If you have not seen this or haven't read it for awhile, slowly read each line of the context driven testing website.

Reading that and giving it thought made my entire day better. If you read that and nod, you aren't alone. I'm there with you, and you are in some pretty amazing company I am learning. Some generous and thoughtful people who help each other get better and continue to learn.

We stand on the shoulders of giants. When I feel alone I discover that I'm not giving enough credit where it is due. In my fear I didn't even reach out for anyone, and they reached out to me of their own generosity and I'm beyond thankful for it. I am entitled to nothing. I try to work hard to earn a spot in the discussion and will continue to do so. I hope that the generosity I've seen from other practitioners in context driven software testing is one day a great investment. Thank you for being everything awesome about software testing. I'm inspired.

The Untrained Tester [Abakas]

I'm pretty much an untrained tester. And yet somehow I generally know how to test stuff. Huh? Okay, so my testing classroom time is limited. I am, after all, an autodidact in test. I'm untrained. But I'm not unlearned. That's an important distinction.

There are many ways to learn how to test something:

Training. Yes, this can work. Classroom or online, both count.
Books. I still swear by "The Complete Guide to Software Testing" by Bill Hetzel. It's a bit outdated in some ways, but it's got a lot of things I can poke at and say, "oooh, is that relevant and how would I apply it?"
Blogs and other online guides.
Google. This one's great for learning specifics, like how to work with certain tools. I tend to hit this one late in the process.
Past experiences. Things we tried that worked, or dev techniques, or things that failed. I learn a lot from coworkers, both testers and developers.

So keep in mind that learning counts, whether its formal training or something a lot more information. If you can to say, "what do I need to know?", then you can go learn it. Don't wait for the formal training. Just go learn.

Testing is Interdisciplinary [Marlena's Blog]

I have 2 undergraduate college degrees. One of them is a BS in Computer Science, the other is B.A. in Interdisciplinary Studies. Here is the program where I matriculated. I had 2 minors as part of this degree, Theater and Art History.

The Interdisciplinary Department at my school was an extension of a residential Freshman/Sophmore program called Watauga College and now called Watauga Global Community. We all lived in the same dorm and had classes together on the ground floor. People outside of our program did not know what the hell we were doing. There were times when we didn’t know either.

We did not have classes like English, History or Art. We would register for 10 hours of an IDS course number. For that 10 hours, we would have lectures and experiences involving topics such as English, History, Art, Religion, Philosophy and Anthropology. For example, we would read about different religions, their histories and then participate in different services of the different religions. I have fasted for Ramadan and was nearly knocked out cold by incense in a tiny Episcopal chapel. After we were finished listening, reading and experiencing we would usually right a 10 to 20 page paper. (Maybe that’s why some of my blog posts are on the lengthier side ;o)

After I decided a career in Theater wasn’t for me, I ended up majoring in Interdisciplinary Studies with a concentration in German Studies. I went to Germany for a year as part of my studies. Deshalb, kann ich noch ein tropfhen Deutcsch verstehen. During my year in Germany, I became obsessed with modern Art History. I dragged my friends to the Stuttgart Modern art musuem several times despite the 3 hour train ride back and forth. When I returned from Germany, I weasled my way into an independent study with a Professor who was rather skeptical that I could write a decent, lengthy paper on Wassily Kandinsky. I made an A.

What is Interdisciplinary Studies?
Another term frequently used in place of “interdisciplinary” is “cross-disciplinary.” This means you are using ideas from different types of subjects, applying some critical thinking and making connections. There is a feedback loop involved. You have to go back and forth between topics refining your ideas and communicating with others to gain perspective on your own thoughts. IDS majors end up with great writing skills because writing is the most mainstream of several ways to work out the crazy thoughts and complicated connections that happen when you put together topics like math, animation, art, and feminist studies.

What does this have to do with testing?
At this point, I believe I have found success as a tester because of my ability to focus/defocus, consider different perspectives and communicate what I have found. The only certainty I have about software testing is that there is no one way to test software. Not only must we understand various aspects of technology, but we have to understand how that technology is applied in a certain subject area for different sets of users, all of whom have a different perspective. This requires technical skills, reasoning skills and communications skills. Because of the rapidly expanding global economy, a global focus on the customer is also required.

How is testing an interdisciplinary?
James Bach’s slide on focus/defocus sticks in my mind. It says that after you’ve been looking at software a certain way for any amount of time, you must pull your head up from what you’ve been doing and violate your own pattern. Focus on the last 4 words. How do you violate your own pattern? To violate your own pattern you must rely on other’s patterns.

Violating a pattern is at the heart of interdisciplinarity. This means we have to find a new perspective on the software we are testing. To do that, it is necessary to understand the perspective of other people different from you. After we’ve done that we must make connections between the different ways we’ve looked at the software. There is a feedback loop involved between the technology we are testing, how we understand that technology and our user’s perspective of how they will use that technology. After that we have to communicate what we’ve found.

There were quite a few IDS majors who self-identified themselves as the disciplinary problems of academia. Teachers either loved us or hated us because we would ask all of the crazy questions and we could spot the bullshit teachers who didn’t know what they were talking about from a mile away. Apathy was checked at the door. Our opinions were often quite different from that of the general college population at Appalachian State and we were constantly arguing amongst ourselves.

I found this quote on the Wikipedia page for “interdiscliplinary”:

Interdisciplinary programs sometimes arise from a shared conviction that the traditional disciplines are unable or unwilling to address an important problem.

Sounds like software testing to me.

This is from The Seven Basic Principles of the Context-Driven School of Testing:

Good software testing is a challenging intellectual process.

As an IDS major, I suggest a refinement for this statement. I think it should say, “Good software testing is a challenging, intellectual and interdisciplinary process.

Great Reasons to Buy Locally...in Canada [PERSONAL FIT MEAL CATERING]

Criticizing Israel Isn't Antisemitism [Tyee - Home]

But a new coalition of MPs seems to say the two are one and the same.

Oh, Mr. Soft [Tyee - Home]

These days, it's more love and less justice for Billy Bragg.

Heads Won't Roll? [Tyee - Home]

BC's premier pledged to cut 'senior executive ranks' by 20 per cent, but can offer no proof it happened. Tyee calculations raise doubts.

The Mother Tree [Tyee - Home]

How fate and fortitude combined to give us a new apple so good it had to be called Ambrosia.

Americanface, Episode 19 [Tyee - Home]

Story so far: After the terrors of Uganda, Britain seems worse.

On the Trail of the Yukon's Black Pioneers [Tyee - Home]

A bit of sleuthing reveals a rich history of risk takers.

Americanface, Episode 17 [Tyee - Home]

Story so far: I leave Tofino and my wife, but run aground in Glasgow

Change your organization, or change your organization [Confessions of an Agile Tester]

That's what Michael Feathers said to me at Agile 2009. It was the end of Patrick Wilson-Welsh's Ugly Code vs Clean Code clinic, and I had gathered a group of people to see if they could help me solve my problem: an organization with a *huge* legacy application (2.5 million lines of *true* legacy code -- no unit testing, no automation, eek!), stuck in a time long since past, where the development team is surrounded by a closely guarded fortress, and the test team was really just there to have someone to blame when customers complained.

After he asked me a series of questions (which really just confirmed that I had already been trying all the things that an expert would have suggested), he said something that stopped most of the group dead in their tracks for a split second: "Change your organization or change your organization". It was a phrase I had heard before, yet at that moment, it hit me like a ton of bricks.

There are countless resources out there for supporting those people who are "change agents", who help spark change where they work, who have gone through or are going through Agile transitions. There are resources for those who are dealing with the more difficult aspects, such as people who just resist change. Markus Gartner wrote up a great summary recently on his blog.

There are many techniques for pushing through resistance, but how can you tell a fight that isn't winnable? Or, at least, isn't winnable for you?

It seems to be something you just "know", from what I can glean. And it is certainly not specific to organizational change. Kenny Rogers sang that wildly popular song The Gambler, "You gotta know when to hold 'em, know when to fold 'em, know when to walk away, know when to run..." The question I have always wondered is "*HOW* do you know?"

I suppose it's not an answerable question at face value. Each situation is different, filled with countless variables, and there is not a single, universal answer.

What is the dark side of what happens to a change agent who is outnumbered, lacking in authority, and just shut down at every turn? Markus's term, "cultural infection" was what happened to me. The crazy culture that I inserted myself into became too much for me to bear, and infected my personal and family life. That was when I knew it was time to fold 'em.

The thing that has gnawed away at me for all of my (relatively short) time as an Agile champion, is "What happens to the people who just don't do well working this way?" or "What happens to the people who don't *want* to change?"

Sometimes, they are pushed out by those who do want to change. I've seen that, people who become so miserable because their interest in working in silos makes them the odd man out, and they eventually get so miserable that they leave.

But what happens when there are more people who prefer to work that way than people who want to change? Or worse (in my view), there may be more (in the organization) who *want* change than those who don't, but the group that doesn't want to change is concentrated in the people who develop the production code?

If an entire team of developers does *not* want to change to a more collaborative environment, or better, to Agile, how can any other department succeed, even if they *do* want to work in a more collaborative way? Can you incent people to change if they don't want to, or have no reason to change?

Believe me, I've tried. I've followed the patterns in Fearless Change, made the countless arguments *for* Agile available on the web, posted slides from various talks on my cubicle, talked to people until I was blue in the face, decided to run *just* my QA department in this fashion, asked the "just try it for a short time" questions, run some small tools on my own that would help the current development dilemma, show graphs and line charts and cost analysis studies, put together presentations. Still, I am forced to sit and wait until a release "in a pretty package" (yes, that has actually been said to me) is ready for my test team to have at it. Still, I get that pretty package, and can't do anything with it because the software doesn't install properly.

I've learned that there are organizations where even though the old-skool Waterfall style of doing things has so clearly gotten them into trouble, they don't see that. Worse, the developers continue to be held up on a pedestal, regardless of those signs. When this happens, the culture of the *whole organization* has obviously supported it, and possibly is continuing to support staying *just the way they are*.

I am reminded of a Jerry Weinberg quote: "Things are the way they are because they got that way".

I'd be willing to bet that even when an organization is this deeply rooted in their ways in the face of adversity, some can still embrace change and come out of it far better than they have ever been. I don't know how to tell the difference between those that can and those that are bound to continue repeating the same mistakes.

I've talked before about the "wall of pain". If an organization never lets those responsible feel the "wall of pain", they will rarely have any reason to change. For me, I've hit my personal "wall of pain" before the organization hit theirs.

Commissioner tables annual Privacy Act Report for 2008-2009 [Canadian Privacy Law Blog]

The Privacy Commissioner of Canada has tabled her annual report on the public sector privacy law, the Privacy Act: Annual Report to Parliament 2008-2009 - Report on the Privacy Act.

At the same time, she has also tabled additional privacy audits, related to FINTRAC and the Canadian no-fly list:

Here's the media release that accompanied the tabling of the reports:

Audits of major national security programs raise concerns for privacy Excessive reporting of personal information to FINTRAC and potential information technology risks with Canada’s “no-fly list” are among concerns identified in audits highlighted in the Privacy Commissioner’s annual report on public sector issues.
OTTAWA, November 17, 2009 — The Financial Transactions and Reports Analysis Centre of Canada (FINTRAC) has more personal information in its database than it needs, uses or has the legislative authority to receive.
This was one of the key findings of the Privacy Commissioner of Canada’s in-depth audit of the independent agency mandated to analyze financial transactions and identify suspected money laundering and terrorist financing in Canada.
A separate audit, also published today, examined the Passenger Protect Program – better-known to Canadians as the no-fly list. It identified several concerns, such as the fact that the Deputy Minister ultimately in charge of who is on the list was not provided with complete information to allow for informed decision-making.
“Since the terrorist attacks of 9/11, we’ve seen a proliferation of new national security programs. We fully appreciate the underlying aim of many security programs – protecting Canadians. However, it is critical – a point reinforced by our new audits – for government officials to integrate privacy protections into all of these programs at the outset,” says Privacy Commissioner Jennifer Stoddart.
The findings of the two audits are highlighted in the Commissioner’s 2008-2009 report to Parliament on Canada’s federal public-sector privacy legislation, the Privacy Act.
FINTRAC Audit
Legislative changes passed in 2006 expanded the types of transactions that must be reported to FINTRAC, as well as the number of professionals and organizations that are required to collect information about clients and to report it to FINTRAC. Examples of entities required to report to FINTRAC include financial institutions, life insurance companies, accountants and casinos.
The audit found that FINTRAC needs to do more to ensure that the amount of personal information it acquires is kept to an absolute minimum. A random sample of files examined in the audit turned up several reports that did not clearly demonstrate reasonable grounds to suspect money laundering or terrorist financing. For example:
A reporting entity filed several reports stating it was “taking a conservative approach in reporting this … because there are no grounds for suspecting that this transaction is related to the commission of a money laundering offence, but there is a lack of evidence to prove that the transaction is legitimate.”
An individual deposited a government cheque for an amount less than $300 and then withdrew the entire amount. The financial institution filed a suspicious-transaction report, but did not indicate why the transaction was deemed suspicious.
A financial institution filed a report about an individual who had deposited a cheque from a law firm. The institution was satisfied that the individual had provided legitimate reasons for the source of funds, but decided to notify FINTRAC anyway because of the individual’s ethnic origin and the fact that this person had visited a particular country.
“It is clear that such reports, containing not a shred of evidence of money laundering and terrorist financing, should not be making their way into the FINTRAC database,” says Commissioner Stoddart.
“It is a bedrock privacy principle that you collect only the personal information you need for a specific purpose,” she says. “The federal government needs to have a justifiable need to collect someone’s personal information. Clearly, FINTRAC needs to do more work with organizations to ensure it does not acquire personal information that it has no legislative authority to receive – and that it does not need or use.”
The audit recommended enhanced front-end screening of reports; stronger ongoing monitoring and review to ensure that information holdings are relevant and not excessive, and the permanent deletion of information that FINTRAC did not have the statutory authority to receive.
Under amendments passed in 2006, the Proceeds of Crime (Money Laundering) and Terrorist Financing Act requires the Privacy Commissioner to review FINTRAC every two years and report the results to Parliament.
Passenger Protect Program Audit
The “no-fly list” is a passenger screening tool introduced in 2007 to prevent people named on a “specified persons list” from boarding domestic and international flights from or to Canadian airports.
The program has sparked privacy concerns, in part because it is secretive in that it uses personal information without the knowledge of the individuals concerned. Moreover, the repercussions for a person named on the list being denied boarding on an aircraft can be profound in terms of privacy and other human rights, such as freedom of association and expression and the right to mobility.
The focus of the audit, however, was to determine whether the program has adequate controls and safeguards in place to protect personal information.
“We were concerned to learn that officials did not always provide the Deputy Minister – who is ultimately responsible for adding to or removing people’s names from the ‘specified persons’ list – all the information needed to make these sorts of decisions,” says Assistant Privacy Commissioner Chantal Bernier.
Other concerns identified during the audit included:
Transport Canada has not verified that airlines are complying with federal regulations related to the handling and safeguarding of the “specified persons list.” The risk of this information being inappropriately disclosed is particularly high for the small number of air carriers that rely on paper copies of the list.
There were no requirements that air carriers report to Transport Canada security breaches involving personal information related to the no-fly list.
Transport Canada did not demonstrate that the application used to transmit information to air carriers met government security standards.
The Passenger Protect Program and the FINTRAC audits, as well as the latest Privacy Act annual report, are available at http://www.priv.gc.ca/.
The annual report also includes details of privacy-related complaints against federal departments and agencies investigated during the 2008-2009 fiscal year. The Office received 748 formal complaints in 2008-2009, down slightly from the previous year. The most common complaints related to access to personal information and to the length of time government departments and agencies were taking to respond to access requests.
The Privacy Commissioner of Canada is mandated by Parliament to act as an ombudsman, advocate and guardian of privacy and the protection of personal information rights of Canadians.
To view the reports:
Annual Report to Parliament 2008-2009 – Report on the Privacy Act
Audit of the Financial Transactions and Reports Analysis Centre of Canada
Audit of the Passenger Protect Program of Transport Canada
Audit of Federal Annual Privacy Reports

Stabbing People with Stuff You Can Get Through Airport Security [Schneier on Security]

"Use of a pig model to demonstrate vulnerability of major neck vessels to inflicted trauma from common household items," from the American Journal of Forensic Medical Pathology.

Abstract. Commonly available items including a ball point pen, a plastic knife, a broken wine bottle, and a broken wine glass were used to inflict stab and incised wounds to the necks of 3 previously euthanized Large White pigs. With relative ease, these items could be inserted into the necks of the pigs next to the jugular veins and carotid arteries. Despite precautions against the carrying of metal objects such as knives and nail files on board domestic and international flights, objects are still available within aircraft cabins that could be used to inflict serious and potentially life-threatening injuries. If airport and aircraft security measures are to be consistently applied, then consideration should be given to removing items such as glass bottles and glass drinking vessels. However, given the results of a relatively uncomplicated modification of a plastic knife, it may not be possible to remove all dangerous objects from aircraft. Security systems may therefore need to focus on measures such as increased surveillance of passenger behavior, rather than on attempting to eliminate every object that may serve as a potential weapon.

How Smart are Islamic Terrorists? [Schneier on Security]

Organizational Learning and Islamic Militancy (May 2009) was written by Michael Kenney for the U.S. Department of Justice. It's long: 146 pages. From the executive summary:

Organizational Learning and Islamic Militancy contains significant findings for counter-terrorism research and policy. Unlike existing studies, this report suggests that the relevant distinction in knowledge learned by terrorists is not between tacit and explicit knowledge, but metis and techne. Focusing on the latter sheds new insight into how terrorists acquire the experiential "know how" they need to perform their activities as opposed to abstract "know what" contained in technical bomb-making preparations. Drawing on interviews with bomb-making experts and government intelligence officials, the PI illustrates the critical difference between learning terrorism skills such as bomb-making and weapons firing by abstraction rather than by doing. Only the latter provides militants with the experiential, intuitive knowledge, in other words the metis, they need to actually build bombs, fire weapons, survey potential targets, and perform other terrorism-related activities. In making this case, the PI debunks current misconceptions regarding the Internet's perceived role as a source of terrorism knowledge.
Another major research finding of this study is that while some Islamic militants learn, they do not learn particularly well. Much terrorism learning involves fairly routine adaptations in communications practices and targeting tactics, what organization theorists call single-loop learning or adaptation. Less common among militants are consequential changes in beliefs and values that underlie collection action or even changes in organizational goals and strategies. Even when it comes to single-loop learning, Islamic militants face significant impediments. Many terrorist conspiracies are compartmented, which makes learning difficult by impeding the free flow of information between different parts of the enterprise. Other, non-compartmented conspiracies are hindered from learning because the same people that survey targets and build bombs also carry out the attacks. Still other operations, including relatively successful ones like the Madrid bombings in 2004, are characterized by such sloppy tradecraft that investigators piece together the conspiracy quickly, preventing additional attacks and limiting militants' ability to learn from experience.

Indeed, one of the most significant findings to emerge from this research regards the poor tradecraft and operational mistakes repeatedly committed by Islamic terrorists. Even the most "successful" operations in recent years -- 9/11, 3/11, and 7/7 -- contained basic errors in tradecraft and execution. The perpetrators that carried out these attacks were determined, adaptable (if only in a limited, tactical sense) -- and surprisingly careless. The PI extracts insights from his informants that help account for terrorists' poor tradecraft: metis in guerrilla warfare that does not translate well to urban terrorism, the difficulty of acquiring mission-critical experience when the attack or counter-terrorism response kills the perpetrators, a hostile counter-terrorism environment that makes it hard to plan and coordinate attacks or develop adequate training facilities, and perpetrators' conviction that they don't need to be too careful when carrying out attacks because their fate has been predetermined by Allah. The PI concludes this report by discussing some of the policy implications of these findings, suggesting that the real threat from Islamic militancy comes less from hyper-sophisticated "super terrorists" than from steadfast militants whose own dedication to the cause may undermine the cunning intelligence and fluid adaptability they need to survive.

Quantum Ghost Imaging [Schneier on Security]

This is cool:

Ghost imaging is a technique that allows a high-resolution camera to produce an image of an object that the camera itself cannot see. It uses two sensors: one that looks at a light source and another that looks at the object. These sensors point in different directions. For example, the camera can face the sun and the light meter can face an object.
That object might be a soldier, a tank or an airplane, Ron Meyers, a laboratory quantum physicist explained during an Oct. 28 interview on the Pentagon Channel podcast "Armed with Science: Research and Applications for the Modern Military."

Once this is done, a computer program compares and combines the patterns received from the object and the light. This creates a "ghost image," a black-and-white or color picture of the object being photographed. The earliest ghost images were silhouettes, but current ones depict the objects more realistically.

[...]

Using virtually any light source -- from a fluorescent bulb, lasers, or even the sun -- quantum ghost imaging gives a clearer picture of objects by eliminating conditions such as clouds, fog and smoke beyond the ability of conventional imaging.

Secret Knock Lock [Schneier on Security]

Door lock that opens if you tap a particular rhythm.

You say you want a revolution … [Matt Heusser's Blog]

Well, folks, I’ve got a lot of ideas intersecting right now that I would like to discuss, but first I have a bit of breaking news.

I’ve worked with several training companies in my day – some independents, some test/quality training companies, development training, and two colleges. Most of the time, the ‘curricula’ consists of ‘whatever we did last year that sold well, plus an idea or two to try out this year.’ (And changing courses at the college level? Good luck with that one.)

Some companies take a few more risks than others, but, for the most part, the inertia of old training materials make it hard for an organization to sense and respond to new events. In the short term, the company is incented to make small changes to existing materials and ends up, like Digital Equipment Corporation, optimizing the mini-computer and missing the personal computer revolution.

But every now and again you get a chance to start fresh.

In the past few weeks I have been approached by several consultants (and yes, one startup training company) for research on what training materials and courses they should offer around skills-based testing. That is to say, not just memorizing a list of words like “Oracle”, “Defect”, and “Fault”, but instead teaching courses with practical exercises designed to yes, help us actually test.

Sure, I can give my advice, but I am just one voice. My experiences as as training customer are limited, and so is my budget. Just about any individual is.

… but the readership of this blog, on the other hand, looks much more like the bell curve of software testers in the world, now doesn’t it?

So I am going to ask for your thoughts and time for that research. If we do it right, we’ve got a chance to materially impact the landscape of software test training in the world today.

The question is simple: Given all the experts on testing in the world, in what way could they organize, package, and sell training and consulting to make it appealing to your business or you personally? What are the barriers to the existing training options?

Example: Flying the team anywhere is expensive. Why aren’t there more local, small trainings in the big cities – New York, Chicago, D.C., Baltimore, L.A.? Or maybe the training should be web-delivered. If it were, how long and intense should it be? Do stand alone course modules work or do you want to be in a class-like environment? What should the core subjects bE?

I think the three-to-five day conference format, with tutorials on the first day or two, is a good thing. I also think that, as an industry, we are leaving behind opportunities to deliver specific training courses – for example, the quick attacks literature is testing gold, and could be an amazingly valuable one-hour course, possibly web delivered. That would certainly be a lot cheaper than a “junket in nantucket”, and offer a lot of value – but I’m not sure if employers would pay for it. And unless people hear that potential customers want it, I’m afraid they won’t develop it.

It’s not for me, at least not any time soon – I’m full-up as is. But when people ask me what courses they should develop, I will point them here. It’s a great time to answer the question “what should we teach?”

Thanks to the recession, I believe we have an opportunity to move the needle on how test training is delivered. Let’s not waste it.

What do you think?

November Software Test&Perf Magazine is out! [Matt Heusser's Blog]

If you like reading this blog, you might enjoy subscribing to software test and peformance magazine, which will give you a half-dozen or more articles and columns a month, all in full color with nifty pictures. If you just want the PDF, you can even subscribe for free by registering for the site.

If you’d like to read our column it is on page 9. The subject is code coverage, an issue that we (especially developers) tend to throw around in a shallow way – hopefully this will put a little more meat on the bones.

If you’d like to have a specific issue – that some might seen trivial – covered in a deep way in STPEdia, drop me a note or leave a comment. Likewise, if you would rather have a single page covering a broad subject in a general way, to hand out to non-testers or new testers, let me know, we can do that too.

Heck, if you’d like to /write/ to ST&P, drop me a note. It might just be your turn.

Tuesday, 17 November

07:45

When life gives you lemons sometimes it also gives you a visit from Gordon Christie. [Testy Testy]

If you have iTunes installed do a search for Edinbus which is available in the UK version as it is for Scotland only. Gordon spent his money to become an iPhone developer and made a free application so that other people in his city can more easily use the bus system. He already has a day job. He is a newlywed, married just over a year ago, yet he wants to do something good for software users. He rode the bus around at all bizarre hours to test his application in many phone states. His application is far better than many of the applications other developers are charging for. Yesterday he showed me the objective C program, which to me looks like a bunch of brackets thrown into a pile and mixed all up, but he assures me it is meant to look this way. I showed him my python scripts I'm working on and we talked about procedural vs. object oriented programming design and why a person would do testing or writing or software development for free. His wife designed all of the icons in his program.

Why is Gordon so wonderful? He is one of the few people out there who gets what motivates me. He wants to work on software that he cares about that provides a wonderful user experience. There are very few companies who understand how important a polished and brilliant interface is. He is willing to spend his time and accept the less money to do so. He really loves good computer software. He asked me not to be hasty or distraught about my layoff, and I asked him to write a blog about his iPhone application development project.

He told me yesterday that I have a year to find a job and I should not just "jump at anyone willing to pay me money". In his opinion, I belong at a company who cares enough about the experience they are providing people with their software to appreciate the talent I have, and not a company who is "in the position that IBM is in or will be in 10 years".

What he is talking about is important for me. For people who are motivated by safety, by comfort, and by ego it is much easier to find a job they would like to stay at. For me it isn't just "about the people", it is about the software I work on. If I am set up to fail or not allowed to positively impact the product quality because of the company culture, I won't be happy there long term. I could lose my passion for testing and be gone from the industry entirely. My second lifelong dream is to drive large construction equipment and it is a life goal for me to drive a backhoe. At least then you really feel like you are making progress. See, I need to make progress or I become dangerous. I have lots of energy, generally positive, but if I feel defeated I'm the sort of person who can cause a minor riot, either for good or evil, so it is important to stay positive. It isn't just about the team, it is about the software being produced. I will put up with quite a load of challenges if I believe in the software.

As my friend, Gordon thinks Adobe is being idiotic for laying me off, but he is sweet that way. I know that due to the economy they made lots of tough decisions. I know that from the fine company I'm in because some really talented developers are also on transition.

This is the bottom line. I'm very dedicated to a positive user experience and high overall quality. I am trained to protect that above all else. I believe I am looking for a job where that is a valued attribute in practice and I am willing to pick up the needed skills in the next seven months in order to be hired on the right team. I am not passing around my resume right now, I am in the information collecting stage. I am very lucky to have some people who are willing to help me land on my feet, and of course, Craig who has got my back, and a long time friend like Gordon, one of the first developers to ever believe in my testing talent to help me brush myself off and get back out there.

What is a test? [Testy Testy]

Version 2.5: A test is an action which produces discoveries that can be used to evaluate product quality.

I'm discovering my limitations with writing. Unfortunately I am not talented with words in quite the way Matt Heusser is. I am trying really hard to learn and sometimes I just close my eyes and think "Learn faster, Lanette!" Anyhow, I hope I'll have another update and I'm still thinking about this.

~~A test is an action which produces data that can be used to evaluate product quality.~~

Why is a test an action? What about those tests that sit in your database? Do they not count? Of COURSE they don't count. They do not become a test until someone or something takes action. They are just a plan. Remember, TEST is a verb. Also, just because you want it to be so does not mean that software is going to test itself.

There was a big battle in my mind over produces. Does it create data? Or does it produce data? Or does it discover data that is already there? Well, the action of the test produces some sort of data I believe. It may not create it, but produces to me also means presents, so it presents it somewhere else. When I am testing I am hunting data to tell me something about the software under test. What about a case where you look at the launched product, notice a button missing and did not have any input or output, but still reported a value bug. To me, this is still a test because there was an action, there was learned information and there was evaluation for the purpose of evaluating product quality. I thought the word data made sense, but then I talked more with James Bach and Ben Simo and the word data was confusing a few other people, and it is very unclear what I mean to say by that term. That word didn't fit in the definition and it didn't convey what I meant either. A test produces more than data. It isn't only inputs and outputs and results that a test produces, but it is learned information. By "discoveries" I mean any information that we have now which we did not have prior to the action of performing the test. The information may have existed, but until I learned of it, it could not be used by me to evaluate product quality.

"can be used" is key. It can still be a test even if it isn't used properly. If I do a series of tests and the learned information is never evaluated, the test still happened because the data exists, but until the data is learned the test is not complete. This happens all of the time with automated tests. The tests happen and the data is never used to evaluate product quality because the resulting data isn't human readable easily and the time to do the translation from the language of "suckish" to human ran out. It still was a test because the data exists, but I would argue that you haven't completed the test yet because the evaluation has not happened. This is a test in progress.

"evaluate" is a tough one. This is where the craft and skill of testing comes in. Taking action and producing learned information is not very difficult. Taking the right action and producing useful data is far more complicated. However, even if you do the first two well, evaluating that data takes skill. I'm not talking about human vs machine here because that doesn't matter at all. Evaluate is all HUMAN all the time, and this is why test automation is deceptively difficult. The evaluation part of a test is when the available powers of observation are used to determine "Is there a bug here?" Answering that question takes into account all of the past experience of the evaluator including their higher level thinking skills, their observation ability, and even their mood at the time of evaluation. There is far too little scientific research done in this space, and I'd like to know what makes the best testers more productive in this space and what factors contribute to their best evaluation versus the times they are just distracted or "off". I can already see the wheels spinning in some executive minds, NO, we can not just automate tests to get around this problem because this data applies to each and every person writing the test automation as well. Until machines can write their own tests, observe, evaluate, and create as well as humans we will have this issue regardless of the method used to take the action of "test".

"produces learned information"-I do not mean learned information in terms of just data, but I mean anything and everything learned during the process of the action of the test. You might have learned that the software is untestable just by trying to launch it, or even sooner, you might have gone to find a build and it didn't exist! So in the process of a step that didn't even state the expectation that a build would exist you've learned something of great value which is relevant to the software quality.

I digress into a daydream here: I wish I knew that there was a team of brilliant developers working on innovative ways to evaluate data resulting from software tests. This to me represents hope for the future of test automation. Finding better ways to validate, collect, and parse the relevant data so that it is more useful for the humans. I think machines could one day test rather than just "checking", but until we program them with more power in the ability to "evaluate" we will continue to just create more automated tests that suck rather than better automated tests. I want to go to a conference next year and see something amazing in terms of how the automated test evaluates the data after the test action.

"product quality"-I include this only because I think this is the difference between a test and random vandalism. When I was about twelve I got curious and tore apart a floppy disk to see what was inside. If I did that with other people's property, I would go to jail. The difference between a test and an act of vandalism lies in the intent. If the purpose is the improve product quality, and I use product and not software because I think this applies to manufacturing tests as well, then the end intent justifies the loss of product. What is the difference between testing and hacking? It is in the intent, not in the action, especially with security testing.

Why do I think this is what a test is and why do I disagree with the IEEE and also with myself? Well, because I was wrong. And because I think they are wrong. The definition they give does not fit the scope of testing. It is far too narrow. I'd love to discuss why the IEEE is wrong and why was wrong before, and if I am wrong now, please, let's talk about it!

One New Thing [Abakas]

One of the amazing things about testing is that you get a chance to try something over and over again. Every release, you get a new chance to try a similar process. Every time you run automation, you get a chance to make it stronger, whether that's per build, nightly, weekly, whatever. We're lucky to have so many opportunities to do roughly the same thing.... better.

Take advantage of that opportunity.

Every time you test, ask yourself what one thing you can do better this time. If you're feeling ambitious and things are generally under control, go for two or three things better. Try a new test technique. Try a new ordering of the test plan to shake out problems earlier. Fix a couple of the reporting oddities in your test infrastructure that have been bothering you.

This isn't news. I've written about changing up your test plan before. It still bears repeating. Do what you were doing... and do one thing better.

Project Doldrums [Abakas]

Sometimes a project gets into a pretty frustrating state, in which:

it's "mostly" done
it's highly visible
it's just starting to get tried by a broad audience, and not all of them know the background and details of the project, just this thing they have been asked to try.

If you're not careful, this is where you get stuck in the project doldrums. Now is the time to avoid getting stagnated on the project. You're getting feedback, which is probably introducing new requirements or ideas. You're probably finding a few issues. It's likely that you have one or two things you already knew you needed to do. And those things just sort of keep piling on each other.

It's now up to you to get control of it and get momentum again. (I could keep the doldrums metaphor going and say that you have to turn on the motor and get out of the listless winds.)

Getting momentum isn't hard, really. There are only three key things that you must do:

Time box it. You should be happy to take feedback, but you're only giving people a set amount of time to provide it. After that, no new requirements, no whining. It will go into production as it is spec'd.
Make your task list public. You have a set of things you now need to do (fix bugs, update config, add a few features). Publish it, and publish where you are on that list. That way you don't get the same complaints over and over, and when it's fixed, you can tell people to try again. It's a way to show that feedback is not ignored, that you will get to it, and that you are making progress.
Do only your tasks. Don't make random changes or other changes. Every change you make should be based on a task in your list. It is imperative that your task list be complete. If you find something else, add it to the task list, then do it. You don't want to give your (now very public) audience the impression that you're flailing around making random changes. It makes them lose confidence in you.

Projects can hit the doldrums. It will happen eventually. Don't worry about it overly; you can get out of them. Just do it with momentum, and do it with confidence.

Rituals [Abakas]

After you work in a team for long enough, you start to develop rituals, formal and informal. A standup is a daily ritual. An iteration retrospective is a ritual. The guy who brings in donuts most Wednesdays is a ritual.

Rituals are great. They are the affirmation that you're a team, and that the team is almost a living organism. It has a heartbeat and habits - and those are your rituals.

But...

Rituals are only affirming if they continue to have meaning. There's no point to having a retrospective if you're no longer coming to small stopping points every iteration. Otherwise it's just a standup, only longer; you can't retrospect in the middle of something. There's no point to having donuts on Wednesdays if you're forced to bring them in; the beauty of that ritual is the small thrill of informality.

Embrace the rituals you have. But evaluate them to make sure there is still meaning behind your rituals. As soon as they lose their meaning, stop doing them. There is no affirmation in empty rituals.

Break It Down [Abakas]

We're working on an internal project that involves (among other things) sending notifications programmatically to Jabber users. It's at that stage where it works for some people and not for others. There are two versions of code doing the sending (different OSes). There are two OSes on the clients, and there are about 10 different clients.

ACK!

So it's time to break it down. It's overwhelming to try to tackle it all at once, but if we make a table we can see what works and what doesn't, and start to try to get all "Y"s into the table, and see if there are any patterns. It gives us a base to work from.

When you're lost, write down what you know, manipulate the data to visualize it, and you'll see a way out.

Sufficient Quality [Abakas]

How do you measure yourself? How do you know your release is of acceptable quality? You've found a lot of bugs, and you've fixed a lot of bugs. You have a set of great new features, and you've done all sorts of interesting security and usability testing. It's a great release! Or is it?

Your release is of sufficient quality if your customers are sufficiently happy.

The real trick here is to define "sufficient". You could have hundreds or thousands of bugs in the product, and if your customers don't hit them or don't mind them (or think a bug is a feature!), then it's still a release of acceptable quality. You could have a total of 5 bugs, but if your customers hit them a lot and they're bad, then this is not a release of sufficient quality.

So if you want to know how you as an engineering (and requirements gathering and sales) team are doing, ask your customers. They're the ultimate arbiters.

Grunt Work [Abakas]

I've been working a bit on some data analytics projects. I've been looking at two major things:

(1) what kinds of issues we find in the field; and (2) what kinds of issues we find late in a release. To do this, I go diving through our defect tracking system. We use Jira, so this is mostly creating filters, and generally runs along the lines of "show me all the issues by client in the customer escalations project".

The problem - and this happens with many things - is that our reporting now has more data than it used to. For example, we didn't used to track which client an escalation was open at as a query-able field (it was just in the text). We now track it as a separate field, but that means that all the old issues were never updated. So I have two choices: I can either construct a special query that pulls the info out of the comments; or I can move the data into the field where that field is not populated.

The advantage to a special query is that I can construct it and I don't have to touch a lot of bugs. The disadvantage is that I have to reuse and maintain that fairly complex query every time I need the information. (And if someone else wants to use it, well I hope they can figure out how!) So instead I'm going to make our old issues comply with our new practices - and populate the field we're now using.

The moral of today's story is:

Sometimes, you just have to do the grunt work.

It's not fun, and sometimes it's more manual than I'd like, but your future self will thank you.

Window of Opportunity [Abakas]

It's relatively easy to decide to change things. It's even fairly easy to decide what you're going to do differently, generally. In order to be successful, though, you also have to consider when you make the change.

Any change has a window of opportunity, time period during which it is most likely to be effective.

For example, let's say you decide to change your release process. Instead of simply sending an email to operations letting them know that the release is ready, you're going to appoint a "development liaison" who will work with operations getting the release in production. The goal of this change is to prevent unintentional misconfigurations (which you've had a problem with in the past). You could do this change right at the beginning of your development effort, but it wouldn't really buy you a lot - after all, you're not releasing so you're not going to try your great new change. No, instead your window of opportunity is a bit before release.

As another example, let's say you're doing iterations and you're not quite perfect at it yet, so the end of an iteration is a bit... frantic. Don't introduce change when you're frantic - it'll only make you more frantic. Your window of opportunity is earlier in the iteration.

So describe your goal, describe your change, and then think of your window of opportunity. All those together will help you gain success.

State Your Purpose [Abakas]

Being a tester, I see a lot of tickets. Some tickets, unfortunately, hang around for a while, and tend to be worked on by multiple people. These wind up with the basic ticket writeup and a series of comments by different people. Particularly when the ticket is a difficult one, there are theories being tried and discarded.

Let's use an example:

We have an issue where access to the system, either for standard use (reading, writing data over mounts) or for diagnosis (logging in to the box) was slow. The system had several exported mounts, was performing replication, and was deployed in our lab. That's about all we knew going in to it.

As we're working on the ticket, a lot of theories came up, ranging from load on the box to a kernel problem to a network issue (turned out to be saturation of the switch when other system using that same switch we engaging in network-intensive operations).

So the question becomes, how do we talk about this in the ticket? There are good and bad ways to write this up.

A Poorly Written Comment

The replication schedule is:

- 20:00 (average duration: 90 min)

- 07:00 (average duration: 45 min)

- 13:15 (average duration: 80 min)

A Well Written Comment

We noticed that the slowness described only occurs sometimes. Looking at what the box is doing at the time, it always seems to be replicating.

The replication schedule is:

- 20:00 (average duration: 90 min)

- 07:00 (average duration: 45 min)

- 13:15 (average duration: 80 min)

We've seen slowness at:

9/12 21:10

9/13 07:08

9/16 07:14

Earlier comments in this ticket indicate that load average is not the problem, but what else might replication be triggering? Early thoughts: increased threads, increased memory use, increased network use...

The Six Month Test

A good comment is one that makes sense six months later, after you've forgotten all the details. This means it needs to:

describe how it relates to the issue as a whole
describe what the reader is intended to do or take away from the comment

Just like your bugs, write your comments for posterity. Future you will thank you.

Paralysis [Abakas]

There are days when I walk into work and have a whole lot of different things that need doing, none of them short. A typical list would look like:

- reinstall an object store (1 hour)

- finalize a task list (45 min, and needs people)

- run a scanner utility against a test data set to gather a baseline (2 hours)

- write up how to do a big configuration I've been working on (3 hours)

- provide feedback on a document (1 hour or so)

And I don't want to start any of 'em because that means I'm not making progress on the others! This is a form of paralysis. Fortunately, it's mild.

There's only one way I know to get out of it, and that's to write down my task list for the day, pick one, and start. It doesn't matter which one I pick, as long as it's one single item.

In this case, I invoked my particular prioritization method:

first the stuff that's blocking other people
then the stuff I'm going to forget if I don't do
then the stuff that others are waiting for but not blocked by
then everything else.

In this case, I did the feedback on the document, followed by the task list (also needed by others). After that, I did the configuration writeup, and then started on the scanner utility (and then the day was over!).

Does this ever happen to you? How do you deal with the "too much to do to even get started" problem?

The Rest of the Product [Abakas]

We have a test plan, and it's great. It covers all the features, and all the workflows of the application. We've got stories, we've accepted them. We've written some automated tests.

Congratulations, you're now half done with the product.

The product is not useful until you can actually use it in production. So now that you've built the darn thing, it's time to think about:

What's production going to look like? How many machines? What configuration?
How are you going to get the software into production?
How about the config info? How're you going to go from dev to test to prod? (hope it's not hard coded into the war file or rpm somewhere!)
Okay, once you got it into production, how're you going to start it?
Come to think of it, how're you going to stop it?
One day you're going to have to maintain this thing. Got a plan for that? Is down time okay? Can you do some rolling upgrades or maintenance?
How are you going to see what's going on? Got logs? Got a way to get logs back to dev for analysis?
How will you know it's running? Any monitoring? Notifications?

There are a lot of questions to answer once you've done the basic implementation. Don't forget to include those when you're thinking about testing it, too.

Taking Notes [Abakas]

I tend to take notes in meetings. As I was doing that today, it occurred to me that there are different kind of notes that I take:

All notes all the time. These are extremely extensive notes. It almost gets everything (but not quite - I'm not that fast). I take these kind of notes when I'm not sure what's going to be important. Typically these are Q&A sessions with customers, or a presentation for which I'm totally unprepared. I try not to do this often because it's really hard to listen and take these kind of notes at the same time. If I'm going to have to share notes with people not in the meeting, these are the notes I take.
Reminder notes. These are much more outline-like notes that I take for most meetings. These are intended to just provide triggers for my memory. These are the notes I take if I need to share them with people who are in the meeting.
No notes. I do this for a lot of meetings. If I need to be actively participating (or leading) a meeting, I generally don't take notes. I'd rather be fully engaged while I'm there.

Do you take notes?

Choosing [Abakas]

We make tool choices constantly, sometimes explicitly and sometimes implicitly. For example:

I write a bash script to grab some network info off multiple machines. Tool chosen: bash. Didn't even think about it, just did it.
We're moving parts of our test plan into Jira. Tool chosen: wiki + Jira. This one we discussed for a while, and eventually made our choice based on some cruft with the wiki. I'm not sure it's going to work, but we're giving it a shot.
I burned a CD of our latest installer. Tool chosen: Disk Utility on my mac. This one is quick and handy, and I haven't gotten a bad burn off it yet.

As we make all these tool choices, we're implicitly considering the properties of the tool and comparing that to the requirements of the task. We have to think about only a few things:

What is the tool good for? Jira, for example, is good for workflows. It's horrible for documentation. A wiki is good for documentation but workflow is simply awful. Some tools are more equipped for long term projects and growth than others. Other tools are a lot lighter and good for quick or small projects.
How convenient is it? The tool I already have will usually trump the tool I don't have, just because of setup overhead. It's not universally true, but it takes a really great feature - or a seriously large annoyance with what I have - for me to switch.
How accessible is it? Whatever tool I use needs to be accessible to everyone who needs it. IMing out info is no good for my boss, for example, who doesn't use IM. If he needs to know, then I can't use the IM tool.

Many times tool choice is a really quick, almost unconscious decision. Other times it takes a lot of evaluation and explicit consideration (especially when it's expensive or has far-reaching ramifications). In the end, though, what tool you choose really only comes down to a few simple questions. So don't stress about it too much. In the end, it is just a tool.

Postmortems [Abakas]

After a release goes out the door, we hold a postmortem. It's pretty standard stuff, usually. We talk about what we did well, what really didn't work out, and what we didn't anticipate.

Timing is an issue, though. You can hold a postmortem right after release, or you can wait to see how it actually does in the field and then hold a postmortem. They each have benefits.

When you hold a postmortem right after you release, you get:

Motivation. People are still stinging from the things we didn't do so well, and are generally aching to fix them. If it was a good release, getting together to remember it will also provide a good boost to people's egos (rightly so!).
Recency. Memories are better right after the release, and you'll be able to have a better discussion about why you did things and what specifically didn't work about them. You'll have a much more precise discussion while everyone's memories are fresh.
Ability to change. If you want to make changes, the sooner you start them, the better chance you have.

When you hold a postmortem after you have some field experience with it, you get:

Perspective. That process ya'll tried that seemed painful right after you did it might not be so painful now. Maybe you've learned it better, maybe you've started reaping benefits, maybe it wasn't as bad as it seemed.
Field Experience. Maybe that release that seemed really shaky has performed like a champion in the field. Perhaps that awesome release ya'll tested extensively has had all sorts of problems. These are things you don't know until it's been out for a while.

My not-so-innovative solution is to realize that our postmortems take us about 45 minutes. That's not very long, so we do both! We hold a postmortem within a week after we release. Then, we hold another one about 6 months later, when the release has been in the field for a while, and we ask ourselves how we really did.

In the land of making things better for ourselves, postmortems are a valuable tool. Holding them twice just lets us learn more from the software and from our development process than we could with just one. Give it a shot.

The Other Fence [Abakas]

A lot of derision has (rightly) been spilled on the idea that development writes code and chucks it over the fence at QA. Fortunately, at least in the places I've worked, this doesn't really happen any more. That development-QA fence is basically gone (hooray!).

Now maybe we should start working on the other fence.

Other fence?

Think for a second about what happens when you're done testing (and developing) on a release. What do we do? We chuck it over the fence at operations (or professional services, depending on how installations and upgrades get done).

Oh, that other fence.

I've been thinking about this fence, and wondering if it's bad. After all, we didn't used to think it was bad that development finished and chucked the code to QA for work! Now we know better. Maybe we should be starting to learn better as we chuck a build out of engineering and into Operations.

Let's posit for the moment that the engineering-ops fence is bad. What kinds of things might we do to break down the fence, and how might that help?

Change how we structure our builds to make them more releasable. This is somewhat analogous to writing more testable code.
Help deploy. Just like our developers do some testing now, maybe we can help deploy, or create some utilities to help. Rake tasks, deployment scripts, hand installations - how does this stuff get deployed, and can we make it easier or better?
Get help building and packaging. Just like development sometimes asks a tester how best to approach a TDD problem, engineering can get some advice from operations on how to handle a configuration issue, or a packaging question.
Pair on problems. When there's a problem in the field, we don't have to look in isolation, or bounce questions back and forth. We can work on it together. With two different views and skills looking at the problem, you're more likely to figure out a problem that has a foot in both worlds.

Depending on your current organization, and on who receives your code, your list may be different. Maybe you're working with support right after release, or sales. Maybe engineering owns operations, so you don't have this problem. At this point, this is just something to think about.

What do ya'll think? Is there a fence after engineering? And is it time to start talking about that fence?

Org Charts [Abakas]

There are a lot of different ways to set up an engineering organization. Generally, they fall into one of two categories: function-oriented, and team-oriented.

A function-oriented structure says that people with similar skills and responsibilities should be a team. That team then provides their collective services to other teams. A team-oriented structure says that everyone working on one goal (project, product, feature area, whatever) should be on the same team.

A Function-Oriented Organization

This is a simplistic example of a function-oriented organization. You have your basic disciplines (development, test, product management) as separate groups, and within those groups you have different breakdowns based on the projects that you do.

Pros:

In-discipline learning is easier and more fruitful. Devs will feed off what other devs are doing, testers will see test innovation and build on it - all because you're working with people who are thinking about the same things you are.
Allows dynamic resource allocations. If you need an extra tester on a project, great, we can add a tester.
Explicit thought leadership. You have a head developer who is explicitly charged with improving architecture and development practices. You have a QA manager who is explicitly responsible for evaluating and refining test practices.

Cons:

People are serving multiple masters. They're trying to help their teams and also to conform to their (function-oriented) organization structure. This leads to some conflicts of interest.
Higher risk of silos. If it's a separate group, then you're more likely to have problems with communication.

Use this when:

You lack predictability in your projects. This happens in consulting a lot, but it can happen in other places, too. If you can't predict how many devs you'll need, it helps to have a pool of devs to draw from.
You have unusual requirements on one or more of your groups. If you're doing some really unusual testing, for example, you may need to keep your testers together so you pick up the learning and innovation effect.

Warning:

Avoid this if you're attempting to embrace SCRUM or some other cross-functional team ownership mentality. The "multiple master" problem will get you in this case.

A Team-Oriented Organization

This is a simplistic example of a team-oriented organization. Each team is a group, and contains members from all relevant disciplines.

Pros:

Unity of purpose. The team is all working toward the same goals. There is no secondary or other goal.
Breakdown of silos. If you can get true team ownership, you start to find developers testing, and testers helping with product management, etc.
No need for functional management. The role of "QA Manager" goes away here. Instead you have team leads.

Cons:

Harder to drive functional change. When you have several teams with a few testers each, it's a lot harder for testers to innovate or learn from each other. The same goes for developers. The groups are simply too small to get that kind of momentum.
Hard to handle changing needs and moving through software development phases. You run the risk of having idle testers as you start an effort, and idle developers at the end of an effort. This is something that can be overcome, but you have to encourage cross-functional work, and be sure to plan appropriately.

Use this when:

You're using SCRUM.
You can have generally stable teams. This implies your projects (or products) are pretty consistent in size and resource needs.

Warning:

Avoid this if you have a particularly weak functional area (or more than one). There's a large risk that isolation within a stronger team will make them even weaker.

So Which To Use?

I've seen organizations of both types - functional and team - work great. And I've seen them both fail spectacularly. The trick is to align your teams with your development and business philosophies. Have you embraced SCRUM? Are your projects generally consistent in size and skill sets needed? Cool - you probably want a team-oriented structure. Do you have highly specialized needs in one or more areas, or an extremely lumpy (in terms of resources wanted) plan? Consider a functional-oriented structure.

In the end, pick the structure that works for you. Just do yourself a favor and pick a single structure. Trying to mix and match will lead to heartache, but pick a single way and you'll give yourself a good chance at success.

Merging [Abakas]

We work in what I imagine is a fairly typical environment. We code away on HEAD for a while. Once we're feature complete, we branch (so now we have HEAD and RELEASE). Then we fix stuff on HEAD and merge it to release until we hit code complete. We also go ahead with the next features on HEAD, but that's not currently the point.

The closer you get to code complete, and particularly after code complete, things get tricky. What do you merge?

There are, after all, several kinds of changes that might be candidates for merging into your branch:

Code changes to production code. Bug fixes, new features, etc.
Code changes to test code. Change the tests, not the code that actually ships.
Infrastructure changes. Change something underlying about the lab or environment (e.g., update the default fstab that gets installed)

So what do we do?

Code changes to production code tends to be the most commonly considered case. You evaluate the risk of the change, how much retesting you need to do, the benefit of the change (and how many of your customers are likely to benefit), and the amount of time left before you really really have to ship. Based on that, you choose to take it or not.

Code changes to test code are trickier. On the one hand, change is change and all change introduces risk. Sure, this code doesn't ship with your product, but it's still change. Plus, you have to consider risk here, too. If your test change breaks something, you might get less information out of your tests in the future, and that would be bad. On the other hand, it probably has some benefit, too: tests run faster so you can do more of them; or a test passes a failure point and lets you expose any other problems that occur later in the test; or perhaps you just spend less time looking at the error that isn't telling you anything new. For me, the bar to test code is lower than the bar to taking production code, simply because the risk to our actual (field) customers is lower, but there are some things that I try to consider:

Is the change caused by a failure or just a cleanup/enhancement/nice to have? The former is more likely to get put in than the latter.
Is the change going to fix something that causes problems for other tests? (e.g., a hang that stops all later tests in the suite from executing). A bad citizen like that is more likely to get fixed.
Is the change risky? The same types of analysis apply here as for product code. Avoid big, sweeping, likely-to-break-something changes.
How many more test runs are we going to have? The closer we get to the end, the longer we can just deal with the problem and not bother to fix it.

Infrastructure changes are generally not really optional. If you want your release tests to keep running in your infrastructure, they have to keep up with changes to your infrastructure. That being said, make sure you really need that infrastructure change, and be mindful of making the changes as small and safe as possible.

Merging is a tricky business, and the closer you get to a release the more of a "gut feel" kind of thing it turns into. So before you get into the thick of it, think about what you will merge and why. It'll save you some arguments later!

All The Other Tests You Did [Abakas]

I've been verifying bugs for the past day or so. It's actually work I really enjoy. The vast majority of the time, it's concrete evidence of the product being better, which is awesome. Plus it's very easy to see the progress I'm making, which appeals to the list maker in me (I love checking things off!).

Here's the thing, though: I'm not just verifying bugs. I'm performing lots of other tests at the same time.

For example, a bug I verified was a display problem with replication progress. This is a small issue, but, hey, it's fixed, so we'll verify it.

To verify it, I had to:

install two systems
create a volume on one system
configure replication between the two systems
configure replication on the volume

So, just to verify one bug, I had to do an installation test, a volume creation test, and a replication test. All I had to do was a quick check to confirm these weren't throwing errors not visible to the end user, and then I have a small other thing done. Repeat enough time, and this adds up to rather a lot.

So next time you're verifying a simple bug, ask yourself what other tests you're doing. You may be accomplishing more than you think!

Today's Top N [Abakas]

I don't drive very often - I live and work within the city, and I tend to take the T to work and pretty much everywhere else, too. Consequently, I don't listen to the radio very often. But this weekend I was on a road trip and I had the radio going. I kept hearing the same theme over and over as I flipped around the stations:

"Today's Top 10"

"the Weekly Top 20"

"Top 5 Countdown"

And I remember thinking, "What a good idea!"

After all, the top 10 songs, or top 20 albums, or whatever, are in some ways like the parts of our test plans:

some of them are the same over and over again: how many weeks is one song at the top of the charts, or at position two or five? Same thing with areas of code.
some of them change each time: eventually a song falls off the list, and eventually we're comfortable enough with an area of our test plan that we move on.
they sound a little different every time: maybe this week it's the dance mix and next week it's the acoustic version - same underlying thing, just a bit different.

So why not? Let's embrace the theme!

We happen to be really close to a release. So for this week, we're having the QA Top 4 (there are four of us, so this makes it easy). Every morning, we come in and pick the four areas we're currently least comfortable with, as a group. We all throw ideas around until we agree on the four. Then we go work mostly on those four items - they're the top of our list for the day. The next day, we repeat the procedure. Maybe it's four different items, maybe some of them are the same and some new - doesn't matter, really. But that's the new QA Top 4. So we work those new four for the day.

The idea here is that we get a chance, every day, to re-identify the scariest areas of the code. And then we work on them. If they're still scary, we'll work on them again the next day. If not, we'll work on the new scariest areas.

There are a lot of ways to prioritize things, but having new ways to think about it sparks new ideas. This is just a new (to me, anyway) way to present that old risk evaluation, and hey, it's kind of fun.

(And by the way, you should see the jokes.... "Replication, by the Again Agains" at #1, and "Defect Verification" off the "Oh Boy It Worked!" album at #2. We're pretty easily amused!)

Good Citizen [Abakas]

As we're testing our software, we have lots of different kinds of requirements. We have use cases, functional requirements, performance requirements, usability requirements, testability requirements, etc. One of the requirements that we don't usually talk about explicitly is the good citizen requirement.

Wait, what's a "good citizen"?

A bit of definition:

Software that is a good citizen behaves in a manner consistent with other software, with regard to interaction with other assets with which it interacts.

That's kind of a pompous way of saying that software is behaving like a good citizen when it does what the systems around it expect (e.g., log in a way that centralized logging tools can handle it) and does create excessive load or resource usage (e.g., doesn't attempt to create hundreds of DNS entries when one will do). In other words, this is software that does what it ought to do, and doesn't behave badly.

Software that is being a good citizen does:

support logging in a common format (e.g., NT Event logs, etc)
use centralized user or machine management (e.g., Active Directory or NIS)
does automatic log rolling
can be configured to start on its own after a power outage or other event
can be disabled or somehow turned off cleanly (to allow for maintenance, etc)

Software that is being a good citizen does not:

log excessively (at least, except maybe in debug, which should be used sparingly)
create excessive traffic on infrastructure servers (DNS, Active Directory, mail, firewall, etc)
send excessive notifications (e.g., a notification for every user logging in would probably be overkill)

Normally, the good citizen requirement is not explicit. Sometimes you'll find mention of it in requirements indirectly (e.g., must support Active Directory for user interaction), but sometimes you won't. You usually won't find the negative requirements (e.g., doesn't renew its DHCP lease too often) at all. But if you miss one and your software misbehaves, you can bet you'll hear about it! Good citizen requirements are generally assumed, even though they're often not mentioned directly.

As you're testing, ask yourself, "is my software being a good citizen?"

"Certifying" Clients [Abakas]

We're a storage company. Lots of people write to us with lots of different programs. These run the gamut from drag-and-drop to homegrown bash script to full HSM solutions, and lots of points in between. Sometimes we'll get asked to "certify a client" application. What's going on here?

Let's break it down:

Who's asking

Usually for us sales or support is asking. Sometimes sales has a client who wants to use a program and wants a guarantee it will work. Other times sales has a client who has not picked a client and wants to know what we recommend. Alternatively, support might have a client who's attempting to use a program and finds something they don't like about it (doesn't work, works too slowly, etc).

Let's say you're not in a storage company (some of us aren't!). This could be a browser, if you're a web app. It could be a reporting or monitoring tool (anyone else ever had a client ask to point Crystal Reports directly at your database?).

Either way, someone's now looking for a guarantee that a client program will work with our software.

"Guarantee"

I'm usually afraid of the word "guarantee". You can do all the testing in the world, and a new patch of a client program will come out and break in a truly spectacular manner. Or the customer will use an obscure undocumented flag you didn't test and... kablooey! (tm Calvin and Hobbes). Or the client will install it on some totally unsupported hardware and scratch his head when it doesn't work. "Guarantee" is a very strong word that means "it's totally my problem to fix".

I usually get around this by saying, "here's what we've tested" rather than "we guarantee this".

Certification Levels

There are a number of different things you can do and call it certification.

The standards approach. This is where you point to some external standard and say, "we conform to this. Any client that works with this will work with us." By external standard, you should make sure you choose a public standard: NFS v3, or W3C compliance, or whatever's appropriate for you. In this case, you don't actually have to test the client. However, you'd better be darn sure you conform to the standard, or this one may eventually bite you.
The "we test this" approach. This is where you offer up the version and configuration you test, and you say that has been tested and will work. Any deviance from that configuration or version may work but isn't guaranteed.
The "certification program" approach. This is where you turn it around on the client application, and offer a certification program. The idea is that they conform to you, rather than the other way around. You offer a set of criteria, test systems (or a lab for people to come test in), possibly scripts and reporting mechanisms, and you let people run your tests. Then you analyze their results, and either put your stamp of approval on or not (think "runs on Vista", etc). If you're large enough and important enough, people will do the compatibility testing for you. This doesn't work so well if you're kind of a tiny nobody in your industry. I've not done this one personally.

So What to Do?

In the end what you do is driven by how much team, time and sensitivity you have. The real goal here is customer (or potential customer) comfort. So you do what you have to do to achieve that customer comfort, within the bounds of the worth of that customer.

My first approach generally is to do a test for that client. If this is an important client, we can get from them (or create if they don't know), a configuration that will work for their situation. Then we test this (and retest it on new versions of our code). It's client-specific, but it gives us the comfort to go to the client and say, "follow our advice and this will work."

My next approach, if this starts to get to be too much volume, is to publish a "known good" configuration (including version) of the client software. We test that client config with every release we do. We tell clients what works, and then let them experiment from there, if they need to.

These two approaches have gotten me far enough, so far. In the end, there's no substitute for trying it at the customer, but short of that, you can give them comfort. And all "certification" really means, at least in this sense, is comfort.

Strict [Abakas]

We're making a big switch in the lab; we're upgrading the underlying operating system on our machines. This is something that we kind of have to do - wouldn't want to be on an ancient OS because it only makes things like security patching harder. I can't say it's my idea of a good time, though - it's a lot of work!

Anyway, once we've done all the basics - setting up FAI, setting up build and test systems, getting the lab migrator to run (so we can move machines from old to new and back again), etc. - then we can start the tests.

We start slowly - one night on the new OS, and then not again for a while. This way teams get a chance to go through their problem areas and fix them before they get hit with the same ticket again. At this point, too, our number of failures is generally quite high, and one or two problems will take out entire swaths of tests ("can't talk to the NTP server: 42 machines can't be cleaned up!" or "stunnel configuration is different: killed two entire suites!"). It's a fairly quick and easy way to find problems that affect each of the teams. It's a learning experience, it makes a bit of a mess, and we all join in cleaning up after it.

At some later point, then, we make the decision that it's time to switch. At this point it's time to get strict. So now we worry about things in the following order:

First, resolve compilation errors. If it doesn't compile, not much is going to run.
Second, resolve bugs that cause machines to leak. If a test causes machines to not clean up after it's done, then they're not available for other later tests. This causes the entire lab to grind to a halt. Generally this is accompanied by cries of "we're out of machines!" and tests just not finishing because there's no machine for them to run on.
Third, resolve bugs that are likely to hide other bugs. If I have a bug in my test setup, who knows what will happen when I get to actually exercising the thing I thought I was testing!
Fourth, handle everything else. Once you've gotten through the first three items, then just start fixing bugs according to your preference.

The overall goal is to expose the bugs. Get things running, then follow up with getting them running right. Hopefully this whole process doesn't take you too long, but sometimes when you're mired in the land of "this underlying thing broke a lot!" it helps to step back and think about what to prioritize. You can make the whole process go a bit more smoothly if you think for a minute, then leap in and start fixing.

Good luck, and happy resolution!

Just Do It [Abakas]

At any point in time, when I sit down at my desk, there are about fifteen things I could do. For example, when I sat down after today's dev leads meeting, I had my choice of:

verify some bugs targeted for the next release
verify some bugs on head
answer a question from a sales engineer
review a potential client sizing worksheet
fix one or more of about three small-ish bugs assigned to me ("fetchlogs doesn't fetch when... whoops!")
read a white paper
work on an in-progress FAI server I'm configuring

In the end, which one I pick isn't nearly as important as one thing:

Just pick something already. Then just do it.

To a certain an extent, what I do doesn't matter as much as simply accomplishing something. I have a general priority list:

active fires
stuff for clients
stuff for sales
stuff other people on my team can't do
other stuff

Based on that, I chose to answer the email first (and do the analysis that I needed to do to provide a good answer). But I could have chosen almost any one of these, and it all would have helped our current situation. And that's the point - just do something. That'll help.

Chatting [Abakas]

I just read this article by Joanne Rothman, talking about a "low level buzz" with chats, emails, and talking going on all the time. She basically says that maybe having a chat open all day, or high email traffic with quick responses might work for some people but not for her.

Our team is colocated, and we still chat all the time. It's a little odd. Basically, most of us have a chat window open on our desktops constantly, and we take a look at it when we're waiting (for compilation, a grep to finish, etc). It's that low-level background buzz. In addition we have, and frequently use, the option of simply walking over and talking to each other.

We tend to use chat for things that the entire group might be interested in, like:

build failure notifications, and their causes
discussion of which branch to run in the nightly automated tests
notification when weekly lunch arrives (the important stuff!)
code review requests
heads up about interesting or risky checkins
pairing requests
general pleas for help (e.g, "how do I do X in perl?")

We get together from there. Someone will wander over and answer how to do X in perl, if it's non-trivial, or will start pairing. It generally works for most of us. I like it in particular because that way you don't have to feel bad about asking questions - you just ask the collective group and whoever is least concentrating at that point, or currently waiting for something, will answer. I'm not then interrupting someone who's really deep in something.

Use for yourself with caution!

Magic Words [Abakas]

Just like there are some magic numbers that can point you to the source of a problem, there are some magic words that also have power. The words you use to describe a problem will color how other engineers (developers, other testers, managers, even you) look at the problem and what they do to track it down.

Just like with the numbers, some of these are common to many systems:

Crash. Most people will be looking for a core file and an uncontrolled shutdown in this case.
Hang. Often you'll get asked how long before you mark something as being hung. Also, this word means that absolutely no progress is being made. If it's progressing at a snail's pace, it's still not a hang; it's just really slow.
Failed. This one means completed with error. It doesn't mean "still going".
Wedged. This is imprecise but generally seems to make people think deadlock.

Other magic words are more specific to your product. For example, in our product we discuss:

Data loss. The system could lose data in this scenario (YIKES!). This one raises red flags all over the place.
Failure. This word means system change that resulted in the loss of availability of a system component. Generally its used for hardware failure (of a disk, node, power supply, network connection, etc). Generally "failure" does is modified by some component name (e.g., power supply failure), and is not used for the more general "software didn't work" case.
Simultaneous Failure Vs Sequential Failure. In a redundant self-healing system like ours, number of failures (see above) matters, as does whether the second one occurred at the same time as the first or after. Depending on which one happened, a whole different debug path will be invoked.

What other magic words are there? I'm pretty sure I've missed some.

Magic Numbers [Abakas]

There are some numbers that I call magic numbers. These are the special numbers that have meaning in the context of your system. These numbers are typically diagnostically important, or triggers to identify problems or potential problems.

Some of these are common on many systems:

86400 seconds: As in, "the test timed out after 86400 seconds". This is a day. As in, the darned thing didn't finish up in a whole day. Oops.
2^32 or 2^32 -1: If you're on a 32 bit system, you're starting to wrap address space here. Look for really large negative numbers where you're expecting positives, etc.
45 seconds: the default connection timeout for network mounts in Windows. If something fails after this long, you're looking at a timeout probably.

Some of these are specific to your system. For example, in our system we know:

6 minutes: The RPC timeout for a query to a remote system is 2 minutes, and it does 3 tries. 6 minute waits mean you're hitting this.
5 minutes: The frequency of heartbeat checks for certain operations. Failover after 5 min means these never succeeded.
25: default max number of simultaneous connections. Can be increased indefinitely, but if you start to see slowdowns or connection timeouts and you have 25 clients in use, you're probably going to want to change it.
XX: Java heap size. (I don't remember this one off the top of my head, but I know it when I see it)

What is interesting is not simply knowing the numbers. What is interesting is the shorthand debugging that it offers you. For example, if support calls up and says that a customer is complaining that the management functions are "very slow to load", and it turns out to be about 6 minutes, then the first place I'll look is to see if it's trying to talk to a remote system, and if there's some sort of problem in that communication.

It's not perfect, but knowing your system's magic numbers can often be a shortcut to finding its problems.

Defining openness: open source, open data, open APIs, open communities, and more [Infotropism]

A couple of weeks ago I was in Florida giving a talk on Open Source, Open Data in which I tried to describe what open data was. In preparation for that talk, I went looking for definitions of “open” as it applied to either field, and found myself drawing on the following documents:

The four software freedoms
This definition of free cultural works, which is based on the four software freedoms
The OSI’s Open Source Definition, and
The Open Knowledge Definition which seems to be largely based on the open source definition.

In the end I structured my talk around the four freedoms because, let’s face it, they’re snappier — but this is all just background.

In any case, I’ve started to collect articles that talk about openness, and in the last couple of weeks I’ve seen a burst of them. Perhaps I’m just hyper-aware at the moment, or maybe we’re going through a phase of introspection about the whole idea. In any case, I thought I’d post a round-up of recent posts on describing, defining, and measuring openness for software, data, APIs, and the communities and processes that surround them.

From the OpenGeoData blog, The Cake Test for determining whether geodata is truly open:

What is the Cake Test? Easy: A set of geodata, or a map, is libre only if somebody can give you a cake with that map on top, as a present.

Cakes are empirical proof that most the data in most SDIs cannot be used freely, because of the licensing terms of the SDIs. And they are an empirical proof that attendants to the latest spanish SDI conference could taste themselves.

Louis Gray, The blurry picture of open APIs, standards, data ownership:

Following the much-discussed news of Facebook debuting its “Open Graph API” on Wednesday, I traded a few e-mails with a few respected tech-minded developers, and found, unsurprisingly, that not everyone believes Facebook is fully “open”. In fact, it’s believed some companies are playing fast and loose with terms that should be better understood.

To quickly summarize the discussion, there are essentially three major ways to bucket “open” APIs…

open access

APIs that leverage open standards
open standard APIs like OpenSocial, OpenID, PubSubHubbub, AtomPub and others

In short, you have “open but we control the process”, “standing on the backs of open” and “truly open”, if this opinion is accepted. The developer adds, “In short, the first two mean nothing, the last one actually fits the dictionary definition. The Web is built on open standard APIs and protocols.”

Simon Phipps, A software freedom scorecard (video from a talk at the South Tyrol Free Software Conference last week) describes why an OSI-approved license isn’t enough to guarantee software freedom, and describes a number of indicators you can use to quantify the freedom of a given piece of software.

Matt Zimmerman, Open vs. open vs. open: a model for public collaboration describes three axes of openness for open source projects:

Open (available)

In order for your project to be open to anyone, they need to be able to find out about it and experience it for themselves.

Open (transparent)

The next flavor of openness is about transparency. This means enabling people to find out about what is happening in your project.

Open (for participation)

The third type of openness is open participation. This builds on transparency by creating a feedback loop: people observe activity in your project, react to it, and then actually change its course.

Finally, Melissa Draper posted about Open community, pointing out external commentary and even criticism is a natural part of having an open (transparent – to use mdz’s term) community.

(Note: Some blockquoted sections above have been edited for length.)

Got any other good links — especially recent ones — on the topic? I’m sure I’ve missed some.

Link [PERSONAL FIT MEAL CATERING]

The local food movement has planted its roots and is growing bigger each year across Canada and the U.S.

Here in Algoma, consumers have been eager to embrace local foods through farmer’s markets, community supported agriculture, community gardens, and businesses such as Penokean Hills Farms.

The Algoma Food Network has worked hard at becoming an invaluable resource to this movement by connecting local food producers to consumers since 2007 and would like to continue this partnership in the years to come.

So to address this growning trend, the Algoma Food Network needs to expand and would like the public to help.

The Algoma Food Network will be hosting a public event, Edible Algoma, on June 17, 2009 7PM at the Great West Life Amphitheatre in Algoma University. At this meeting, the Network will be presenting its current priorities and will host an open forum. The network is also announcing a change in leadership, with the appointment of Birgit Kroll as chair. Lee-Ann Chevrette, the former chair, has recently moved to Thunder Bay.

The Algoma Food Network is dedicated to building and maintaining a network of local food producers and consumers through education, advocacy, action, and relationship building.

The Island Where People Live Longer [PERSONAL FIT MEAL CATERING]

Making it to 90 years old is awe-inspiring in much of the world. But on a tiny Greek island in the North Aegean Sea, nonagenarians barely merit a second glance.

The island of Icaria could be the newest of the world's so-called blue zones — places where the people have unusually long life spans.

Read more on Mark Bittman here

Seedy Saturday, 2009 [PERSONAL FIT MEAL CATERING]

Seedy Saturday
Sault College-Native Centre-G Wing
Saturday February 28, 2009 10:00-3:00
Co-hosted by Clean North, Sault College and Seeds of Diversity Canada. Come swap or purchase heritage, open-pollinated and/or unusual seeds from local gardeners. Visit local environmental, food security and gardening organizations and vendors. Speaker presentations run from 11 am to 2 pm. on a variety of topics such as seed saving, germinating seeds and guerrilla gardening. Don't miss Kate Green from USC's Seeds of Survival program who will be showing 'Hijacked Future' and speaking about food sovereignty and international seed saving efforts. Those participants wishing to participate in the swapping of seeds can register from 10 am to 12 noon for the exchange that will take place between 12:30 - 3 pm. Refreshments, children's activities and lots of handouts. Fully accessible site. Admission by donation. As an added bonus, Cinema Politica will be hosting the viewing of 'King Corn', a feature documentary about two friends, one acre of corn, and the subsidized crop that drives our fast-food nation at 7 pm in the Multimedia Centre located in the B Wing of Sault College. A panel discussion will follow.
Contact: Suzanne Hanna at (705) 759-2893 wildgardener@shaw.ca or Laura Wyper at (705) 253-1951 laurawyper@yahoo.ca

Tips on Winter Eating [PERSONAL FIT MEAL CATERING]

The last thing I want to eat when it's -25 is a salad...especially not as a meal. Soups and stews with fresh baked bread are what keeps me going at this time of year. It is a challenge to balance diet and lifestyle after the holidays when the cravings kick in, not only for sugar and fat but also warmth and sunshine. There are ways to keep up your energy even if a stay in a warmer climate is not a possibility. In today's Globe, Leslie Beck has some good advice about keeping this winter a healthy and happy one.

read more here

File under interesting... [PERSONAL FIT MEAL CATERING]

Some half a million people in the United States experience seasonal affective disorder (SAD), according to the American Academy of Family Physicians. Symptoms of the condition, also known as winter-onset depression, include anxiety, fatigue, and irritability, and the problems may keep coming back every winter.

The disorder is thought to be caused by the lack of sunlight that some people experience during the winter. It also may be an evolutionary remnant of hibernation according to columnist Carol Venolia in Utne Reader’s sister publication, Natural Home magazine. As recently as the early 20^th century, Venolia writes that peasants in both Russia and France would shut themselves in for the cold months, huddling around the stove and barely moving until the spring thaw.

Venolia advocates giving into our hibernation tendencies, at least a little bit. If we did, “We’d sleep more and demand less from ourselves. We’d be more inward and reflective.”

Bye Bye Banana [PERSONAL FIT MEAL CATERING]

The modern banana, one of the world’s most popular and troubled fruits, may be on the brink of extinction. Though its shape is evocative of sex, the banana is “a sterile, seedless mutant—and therein lies a problem,” Fred Pearce writes for Conservation.

In the wild, bananas are basically inedible due to their hard seeds. Most bananas found in grocery stores are “mutant plants” and “sterile freaks,” Pearce writes. They're almost all of just one variety, the Cavendish, after its predecessor, the Gros Michel, was ravaged by disease. Today a new disease is stalking the Cavendish, and experts believe it could be just a matter of time before the modern banana goes virtually extinct.

While resistance could be bred in other types of fruits, Pearce reports that “Because all edible varieties of bananas are sterile, introducing new genetic traits to help cope with pests and diseases is nearly impossible.”

For the time being, the disease is kept at bay by massive chemical sprays. “Forty sprayings of fungicide a year is typical,” Pearce writes, “making the Cavendish the most heavily sprayed food crop in the world.” This has led to a variety of health problems in banana workers, including sterility and alarmingly high rates of leukemia and birth defects.

Some are looking to genetic modifications as a last effort to save the troubled banana, but that raises a host of other ethical and environmental problems. And whether or not the general public would eat a genetically modified banana remains to be seen.

Americanface, Episode 18 [Tyee - Home]

Story so far: Stuck in Glasgow, I recall my African genesis.

Americanface, Episode 17 [Tyee - Home]

Story so far: I leave Tofino and my wife, but run aground in Glasgow

Returning Fire on Gun Registry Critics [Tyee - Home]

Last week, many Tyeesters assailed my defence of the registry. Here's why they're wrong.

Who Gets to Own a Telecom Firm in Canada? [Tyee - Home]

Upstart Wind Mobile wants to compete with the Big Three. But foreign investment rules might stop it. Is that old-fashioned?

Where the Wild Things Were [Tyee - Home]

Children of divorce, loneliness and other magical things.

SFU Profs Slam Campbell's Energy Plan [Tyee - Home]

Hobbling BC Hydro so private firms can profit big is bad public policy.

'A Woman among Warlords' [Tyee - Home]

Afghan author and firebrand Malalai Joya on Canada's mission, Obama's Nobel, dodging assassination, and more.

Americanface, Episodes 13-16 [Tyee - Home]

All this week's episodes rolled into one.

Analysts Argue over Olympics Effect on BC Tourism [Tyee - Home]

New report puts impact at zero so far, but another expert says 'more clever' counting yields rosier view.

It's still a *bug*, and your denial does not change that [Confessions of an Agile Tester]

UGH!! I've had this frustrating discussion with a developer recently that is really annoying the heck out of me.

In the particular scenario being discussed, this product has a history of not installing correctly. It's a (mostly) .NET desktop suite of programs with a central processing engine. The most common installation errors involve components not registering correctly. The crazy thing about that type of issue is that it manifests itself, for this particular product at least, in fairly inconsistent ways. Sometimes, a user function doesn't work (while sometimes it does!). Sometime, the processing engine locks up unexpectedly.

In the past 4 months, I am certain that I have seen the number wasted man-hours troubleshooting something that ended up being an installation problem climb well into the hundreds. In QA alone, I can cont at least a week, times 3 people. I'm already at 120 -- not counting support, developers, product management ....

There was a customer issue that they had spent the past 2 weeks troubleshooting, only to find out that a file had not been registered properly. So product management asked the developer the following:

Is there an identifiable bug in the installation code that we could track in Bugzilla for inclusion in a future release? Or is this issue a one-off?

And the developer's response was that there was no bug in the installation code, so there was nothing to fix. He went on to explain that the files has just not registered properly or were corrupt. He said that uninstalling the suite should have cleared the problem. (I don't want to copy his response, just because it's fairly product-specific.)

So this was my response to that (I was actually fairly nice in it, because seriously, it drove me *crazy* to see that "it's not a bug" statement!)

Though I can see where there is not a clear bug in the installer, I don’t think that is the same as “we are missing something and should implement some checks”.

It just really seems to me that *any* install that we do should be checking itself, that it installed correctly. I have seen dozens of man-hours wasted in just 4 months, on what came down to things not installed properly.

“It didn’t tell me that it didn’t go right” is a bug, in my book.

Now seriously, let's look at this issue.

If the software does not install properly, that's a bug, right?

If the software fails to install properly, AND does not indicate to the user that *anything* has gone wrong, that's a bug, right?

If this install problem causes the software to not work, that's a bug, right?

Is it considered okay to ask the customer to uninstall and reinstall their product (no trivial task, mind you) when it's not working? Okay, anybody but Microsoft?

Has 10 years of testing led me way far astray from the basics? Am I way off base here?

Lanette Creamer suggested that the developer was right; that it wasn't a "code bug", but rather a requirements failure. This is a valid point, but for some reason, I hate having to "frame" things in a certain way to placate a defensive person.

The bottom line is that this product is doing something it should not be doing, and it is impacting the customers, the developers, the support people, the QA people, the product people ... IT SHOULD BE FIXED. Seriously, don't throw blame, don't deflect attention away from yourself.

Let's instead sit down together and figure out where the problem is and how to solve it.

Harmony is beauty [Confessions of an Agile Tester]

First and foremost, if you read my blog. I have to suggest that you go grab yourself a copy of Beautiful Testing. It's a great read, and has inspired this post.

I've been inspired by reading this book. I've been reminded of my passion, of the reasons I have chosen to do what I do. After all, I chose to be a tester, I haven't just fallen into a job and decided I should deal with the hand I've been dealt.

I didn't always want to be a software tester. As a young child, I wanted to be a marine biologist! Actually, I wanted to study the neurological systems of dolphins. I was convinced that I was going to figure out what it was about their brains that made them so smart. I still have the book that inspired me then, 9 True Dolphin Stories.

After a marine biology course and choking through learning the ocean zones, I shifted my interest over to genetics. There was something about genetics that was *beautiful* to me. I was amazed at how the whole system *worked*. Granted, sometimes it goes awry, but at an incredibly rare rate. I liked it because it is so *simple*, and yet so completely *complex*. I was amazed at how 4 simple bases could turn into a *human being*. (HA! In the middle of writing this blog post, I went to Barnes and Noble and happened to come across a copy of Matt Ridley's "The Agile Gene"! I'm a Matt Ridley fan, and this title is a re-title, but very coincidental timing!)

When looked at that way, genetics and computers are rather easily related to each other. In genetics, a 4-letter alphabet turns into some complex stuff, like human beings. In computers, a 2-letter alphabet turns into some complex things, too, just not as complex, and too often, not as smoothly. So in a lot of ways, my switch from genetics to computer science was an easy one.

When I work toward making software beautiful, I have in my head a picture of how it's done in genetics. Those systems are exponentially more complex than software, and yet, there has somehow developed a *synchronicity* (Is that a real word?), where even given countless variables, the system is responsive and *works*, regardless of the state of all of those other variables.

Isn't that incredibly beautiful?

Reading Beautiful testing, I saw these same ideas threaded through the chapters.

Matt Heusser talked about math in terms of number theory and proofs, including some commonalities in nature (the Fibonacci sequence and pi, for example). In studying mathematics, Matt found an appreciate for what he calls "aesthetics" (and what I've likely been calling 'harmony'). His sentences on the aesthetics in mathematics hearkened back to my appreciation for genetics for the same reasons.

But it was Chris McMahon's chapter, relating creating software to an artistic performance, that inspired me to think about the way I work on a software team in a new light. He mentioned the way a band rehearses together before a performance. I have seen some *amazing* bands, where I watch them communicate with each other silently. The slightest gestures can be made, some not even perceptible to the audience, and the rest of the band follows suit.

These things happen all over the place! A new mom once told me that she enjoyed me coming over because I "just knew" what she was going to need before she did. Well, of course I did, I've been through the mom thing before. I used to ride horseback (dressage, actually) and with some of the horses I trained on, the *slightest* shift in my body weight caused the horse to change direction. I had to learn to control every muscle in my body to only send the messages I intended. Have you ever seen a couple, who at a party, seem to be able to send signals to each other with just a glance of the eyes?

Coming back to testing and software, it seems that the very best teams work in much the same way. They are able to shift direction, together, without falling out of step with each other. They are able to fill in for each other without the need to go through hoops to do so, and they trust each other completely. They all care about the same things, and work together to accomplish their goals, following the same values.

As a tester, I love the fact that I get to strive toward this kind of harmony, while also managing the effects of so many other variables: the customer, usability, risk, time, cost, value, technology, the business goals, product management, etc etc. For me, being involved in testing means that I have my hands in everything, and spend my time trying to make sure that they *all* work together seamlessly. As a tester, I must represent the customer. I must represent the potential customer. I must represent the developer. I must represent the business. And, I must tie all of these together in an intricate web, ensuring that the outcome represents all of them and is responsive to any variation in their needs.

THOSE are the reasons I am drawn to Agile. THOSE are the things I see in the teams I have had exposure to that I think work *really well*. THAT is why I love testing. Testing *is* beautiful.

The difference between motion and action [Confessions of an Agile Tester]

I just read this amazing blog post by Steve Blank, and felt compelled to share it.

I'll just post a link now, though I suspect my own spin on it may be coming ...

The Difference Between Motion and Action

Making Quality a Priority [Confessions of an Agile Tester]

Here is a quick PowerPoint presentation I think may be useful to some other people, too.

The intended audience is business people or executives who still need a little bit of understanding in what it means for "quality" to be baked into every part of software development, from inception through to release, and in what it takes as a team to have the best collaboration between developers and testers.

Please feel free to use it, modify it, etc. And as always, if you have feedback, let me know!

Making Quality a Priority

What if you could design your ideal "software creation" plan? [Confessions of an Agile Tester]

So, a special situation has been presented to me, and I'm going to try to solve it in a collaborative way. It's short notice, but I have what I have, you know?

I've been given an opportunity to develop out a roadmap for ensuring that our very LARGE legacy product heads in the right direction from this point out. Their current focus is apparently centering on the word "stability" Roughly, stability, to them, means that what we have designed our product to do, *works* and is satisfactory to the customer (things like performance cannot be ignored here, for example).

This team (people/process/infrastructure) roughly works on the level of a bunch of unorganized college kids, so there is a lot of room for improvement. They need some fundamentals put in place, for sure -- Automatic builds, CI, unit testing ... (clearly the list goes on from there). Although I specialize in testing, I believe that our executives now understand that "we can't test quality into our product" -- we have to be doing things with quality from the beginning. As I think about all the things this team needs in order to be working in a fashion that does actually move them forward, I want to be sure to optimize such a transition.

I want to hear from others, however, about what *they* would do, if given such an opportunity. I want to brainstorm as a group how people *wish* their teams worked. Let's draw up an ideal "software creation" plan -- beginning to end, all of the pieces.

I have a space available for a physical location, for those who are local. Please send me an email (address at the top right of the blog) for specifics.

I'll be offering a remote-in ability, too. I've created a GoTo meeting that offers both VOIP connection and phone-in connection.

I'll tentatively plan on holding this at 7:00pm EST today, Thursday, October 22 (Sorry! I meant the 22nd!).

1. Please join my meeting.

https://www2.gotomeeting.com/join/604606915

2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone.

Dial 630-869-1014

Access Code: 604-606-915

Audio PIN: Shown after joining the meeting

Meeting ID: 604-606-915

Getting Started with White [Confessions of an Agile Tester]

I've recently decided to dive into the world of Project White and MS UI Automation, since this is relevant to my current project. I had to create a tutorial for a team to get set up to use white to begin with, so I decided that I could share that tutorial with everyone else as well. It goes into some detail about Visual Studio as well, and is intended for an audience that has not necessarily done automated testing through Visual Studio before.

The tutorial uses MS Visual Studio, C#, White, and UI Spy to create an *extremely* simple test. My intent was just to show how to set it up. Next time, I will describe starting to write tests, including common functions in White and writing good tests.

Please, please offer feedback. I would love to be sure that I am doing things the best way possible!

White Tutorial

*be the worst* [Confessions of an Agile Tester]

Huh? Did I just say "be the worst"? Yep, I sure did. But before you go telling your boss that I told you to be the worst tester you can be, let me finish the phrase!

First and foremost, I have to give credit where credit is due. I saw this phrase first from Chris McMahon -- he says that this meme has been around the music business for a long time. He credits Pat Metheny, who said, "...try to be the worst guy in whatever band you're in. That's the secret."

Given the context, what I am saying is "be the worst of the people you are surrounded by", or "surround yourself with really great people."

At Agile 2009, someone told me that Elisabeth Hendrickson decided to learn by inserting herself into the best teams she could find. Even before I encountered the "be the worst" quote, I had begun bouncing around the idea of how I perceive myself versus those I work with.

I think back to an early job in my career, straight out of college. At this place, I remember thinking to myself with some frequency, "These people are *SO* *SMART*! I feel SO DUMB when I am around them!" I often tried to just keep up with conversation, hoping to fake it long enough to avoid appearing dumb, too! Looking back, how I wish I could have gotten over my own insecurity and taken the opportunity for exactly the opportunity it was! What I should have been thinking was "Wow, these people are *SO* *SMART*! I want to learn everything I *can* from them!"

What can we get out of "being the worst", and why would anyone suggest that?

I think we can get a *lot* out of it: learning, experience, growth ... In working with people who have a set of skills that you wish to expand for yourself, you can see first-hand, on a day-to-day basis, and under a whole slew of circumstances how that quality is manifested. Sometimes it might be a technical skill. Maybe it's a communication skill.

As I have grown in my career, I have begun to feel like the "dumb one" less and less. I think that my passion to get better and better at the things I want to be good at, have made it more and more difficult for me to *be the worst*. What have I done in response?

I've become *way* more active in the agile community. In that way, I can surround myself (though not as frequently as I would like) with those people I see as *way* more skilled than I am in certain things. I found this out with certainty at Agile 2009. I love talking to Elisabeth Hendrickson for her insight into agile testing and human relationships at work. I enjoy Lisa Crispin's company for her amazing ability to be a great agile tester, without falling back on programming the hard things (like I do!). I met Patrick Wilson-Welsh, and admired his passion, sense of humor, and ideas on good, clean TDD. I had conversations with Michael Feathers and Bob Martin to try to gain some insight into my specific legacy code issues. I had great conversations with Antony Marcano and Andy Palmer about testing tools and frameworks. I *pair programmed* with Abby Fichtner to gain from her development experience. Of course, there were many others ....

In this way, I forged relationships and surrounded myself in a way that I could *be the worst*. I'll keep saving my pennies to go to conferences and keep being active in the agile community so that I can keep *being the worst*.

Do others try to put themselves into situations where they can *be the worst*? How do you keep yourself always learning and always surrounded by those you can learn from?

Does having a separate maintenance team hurt efforts toward accountability? [Confessions of an Agile Tester]

I was recently thinking about a situation that puzzled me, and came to a realization that makes me a bit nervous ....

I have had exposure to some teams where even though they are in firefighting mode more often that is comfortable, and/or even though their customers are unhappy, and/or even though their software has a *lot* of bugs, they still don't seem to "get it" and try to do things in a better way.

I have sat, baffled, looking at people who lead these teams, my jaw on the floor, wondering why the people leading these teams are passing the buck on the team's responsibility for problems, and instead presenting a magic show to prove why the problem is not theirs.

And today, something hit me: I was thinking about what would cause someone to change; what would make it so that they finally felt the need to do something different, and the term "the wall of pain" came to mind. This term comes from a blog post I read a few months ago, "Fighting Technical Debt with the Wall of Pain". What would instigate change? Enough pain that it feels necessary, in order to escape the pain.

This reminds me of Parenting 101: How do I stop my 12-year-old from making irresponsible decisions? I make them painful enough that he will be sure to avoid the pain. Luckily, these days that is as easy as "Oops! Your PSP ran away! I can't find it!", or "Where are your Playstation controllers? Gee, I don't know ... I guess maybe they will return when you start doing the chores that you are responsible for."

BUT ... BUT ... what about companies where the new development team is a separate entity from a maintenance team? What if the developers who are writing new code are sheltered and protected from the results of poorly written code? In some cases, it goes something like this:

- Development team writes software ... Maybe they fix a few bugs along the way. Sometimes, if bugs come up and it can be proven that they existed in previously written code, they may shrug them off as not the result of current work, so not their problem right now.
- Development team's schedule gets squished, and as it gets closer to release time, only the *most* important bugs that are proven to be the result of their current work, are even looked at by the team.
- Release happens. At this point, product is a released product, and the workflow now looks like this:
- Customer has an issue, customer calls Support. Support tries to reproduce it, and if they can, they file a bug. Bug gets reported to Maintenance team, and they work on it.

In the above scenario (which exists in some companies), the team that originally wrote the aforementioned poorly written code never sees the wall of pain. Seriously, for the most part, there is absolutely zero repercussion to them directly for bad code.

In a case like this, the responsibility for well-written code falls upon those who are internally motivated to do their best, and strong enough personalities that they fight the push to get *something* out *faster*, well-written or not. I believe that this type of personality does not describe the majority, and the whole team falls into a pattern of continuing to write poorly written code. What is there to stop them?

Has anyone had any experience helping a team to transition into a more agile-like process from a situation that looked like this one? How do companies like this adjust to fight this pattern?

I wonder if it would help to break up the idea of "new dev" versus "maintenance" teams, and instead cross-pollinate into functional teams. In this case, there would be a team for one specific module, component, or module, and they would handle both new work and bug fixing/maintenance. Would this strategy work?

Work to make a company successful, or work for a successful company? [Confessions of an Agile Tester]

Once in a while, I think we all find some statement we read somewhere, while doing something, that sticks with us. I have *definitely* found that to be true very recently, mostly because it puts into words a concept that has rolled around in my head for quite some time.

First, some background .... If you read my blog, then you know that I have been known to rant, at times. Really, I try not to, but it happens. Sometimes, it is just that I want *so badly* for others to change, and I have a hard time understanding why people don't just *do* things the best way, all the time. Often, when I find others to talk through the issues I am encountering, they ask me questions like "Why do you still work for that place! It's hopeless!" (Sometimes, I believe it is, and I leave, as I did in 2008.)

Sometimes, however, I stick around for a while, and I have a hard time explaining why. And then, recently, I saw a sentence that resonated with me.

Twitter was all abuzz recently with responses to Joel Spolsky's recent blog about "The Duct Tape Programmer". He quotes Jamie Zawinski, and Elisabeth Hendrickson (@testobsessed) pulled out Zawinski's resignation letter from AOL.

This one quote resonated with me, and continued bouncing around in my head:
"...you can divide our industry into two kinds of people: those who want to go work for a company to make it successful, and those who want to go work for a successful company."

It rolled around in my head for a few days .... Did I agree? Did I think that people could generally be so easily divided up into just those 2 groups? Probably not quite *that* easily, but for many people that I have encountered in my career, I could place most of them into one of those pretty quickly.

For a while, I have described these groups in the following way: people either seem to come to work because it's a job, and they do their job, and that's it, OR they seem to be really passionate about *what* they do, and are constantly striving to make things better. I fall into the latter of those 2 categories. I most certainly do what I love and love what I do.

But Jamie's sentence looked at things from just a slightly different point of view, and after thinking about it for a bit, I believe, describes *me* just a little bit better. I have *always* wanted to work to make things great, and not so much wanted to work for/with/on things that were *already* great.

This can be applied to many aspects of my life. I was the single mom who put herself through college, working full-time and taking full-time classes in CompSci. I am the vocal advocate of a high functioning autistic child who wrote to the local newspapers when the school district was failing at its job. I chose to bring a Siberian Husky into my home (this sounds silly, but Husky owners will tell you .... WOW).

I THRIVE ON THE CHALLENGE.

That's it, that's me. That's why I like testing over development. I have said for years that developers have to find one way to solve a problem and testers have to find ALL ways to un-solve the problem. Testing gives me the challenge of being creative, being technical, being analytical, speaking in at least 2 different languages, juggling an obscene number of balls in the air and being squeezed and sometimes disrespected along the way. But, it's a challenge, and I like a challenge.

In the same way, companies that are struggling, companies that don't "get it", and have fallen into the proven patterns that destroy great ideas, are a CHALLENGE. They are high-risk, for sure, and sometimes they offer little reward other than knowing I am doing my best, but *if* I can affect change, *if* I can help, the reward is incredible.

I believe that for "successful" companies (I place "successful" in quotes because this is a relative term .... for me, in this context, it is mostly about doing things the best way possible), I would personally gain little reward because there would not be *enough* of a challenge.

I wonder, then ... how many other great testers that I know are like-minded? Do other people enjoy testing because they enjoy a challenge? Would they generally choose to work to make a company successful, or choose to work for a successful company?

What do you do when ... ? [Confessions of an Agile Tester]

What do you do when you are "just a QA person" or "just a tester", and you *know* *know* *know*, down in the deepest depths of your coding soul, that the developers on your team are DOIN IT RONG?

Do you quit the job and go find some decent coders? Nah ... What fun is there in that?

Yesterday, I attended Agilepalooza 2009 in Charlotte, NC. There were 2 main tracks, one for learning about agility with several speakers lined up and one for "advancing agility", which was Open Space format. David Hussman was there, and so was Jeff Sutherland, with whom I got to have a great discussion about testing on an agile team. It was a VersionOne sponsored event, and yes, it was myself who yelled out during the introductory meeting "Does anyone use Rally?" You don't have to believe me that I honestly did not know at that point that V1 sponsored it, though I sure played on that point later.

ANYWAY, I digress. I *love* going to conferences, especially those that allow for the opportunity to have in-depth discussions with others out there "in the trenches". I get some practical knowledge out of talks, but I get the *most* out of talking to individuals, and being able to say "I am struggling with this particular problem right now ... have you dealt with this before, and/or do you have any suggestions?" I have found that most of the Agile community, and especially the testing-focused subset, are always very willing to help and share and ask for your help right back.

So I had the opportunity to ask a few people my most pressing "What do you do when ... ?" questions, and here are some of my highlights.

Q: "What do you do when you know that the software you are developing against is unstable and likely not even following normal standards?"
A: Try Fxcop (I'm working on .NET apps), or FindBug for Java. WOW! I have actually heard of FindBug before, but haven't worked that deeply on Java programs in some time, so I have not spent much time with it. I got an opportunity to check out the fact that Fxcop has a whole type of messages centering on Performance, which happens to be a specific focus for my current project. I CAN'T WAIT to be able to use it.

Here's where this answer gets creative though. In true "don't pull punches" form, I would be the type to toss all 8,000 errors (arbitrary number) at my dev team. Instead, it was suggested that I pick a SINGLE error, print it out, and take it to a developer. I can tell them that I was using this tool (Fxcop), and here was an error (relevant to current work) that might help our current efforts. I love the idea, and I don't think I would have ever thought of it on my own. It is so simple, yet so ingenious .... I'll be doing this one soon.

Q: "What do you do when you are having trouble automating tests around custom controls that require some specific events to be called?"
A: Grab a network sniffer and watch exactly what events get called in the background! In this particular case, the suggestion was to use Knoppix (a lightweight Linux distro), which has a sniffer built in, or Chris McMahon suggested wireshark.

So, my background on this question is that I have been trying to throw some automated tests onto our web client, and we use Infragistics controls. I know that IG (Infragistics) uses Webaii internally, but still have been unable to get through exactly what I need to call and in what order to get Webaii to interact with these guys. The disappointing thing is that Telerik, who released Webaii AND who also makes custom controls (RAD Controls), posts right in their forums and on blogs about how to automate their controls. Infragistics does not. I have tried many permutations of interactions with these controls (my immediate need has been a WebCombo and a button), and have eventually had to put them on a back burner for other tasks.

However, if I can get past that one hurdle immediately, I can move forward with starting the seriously overdue task of putting some automation around our software.

Ok, those are my 2 biggest takeaways from yesterday, and I am sure there will be more later .... For the record, I have to give credit to Jared Richardson for the ideas presented above. It was really great to meet him and to talk with him about some specific issues, even if he did almost fall off the table listening to me :)

Help! My Selenium tests are flaky! [Confessions of an Agile Tester]

In my development of Selenium tests, I have come to dread the oh-so-common "Permission denied" errors. A quick scan of google search results indicates that there is likely some cross-domain issue, and a suggested workaround is using an experimental browser, like *iehta. I generally use *iehta anyway, and still seem to have problems with the "Permission denied" error.

In my most recent experience, testing against what seems to be an inconsistently slow and sometimes flaky web app, I have come to expect that "Permission denied" usually indicates that an element that I am trying to access is not yet accessible. For my app, which is heavily AJAXed, "waitforpagetoload" doesn't cut it. Since the AJAX scripts are still running, Selenium would wait until the end of time (or the timeout, whichever comes first) for the page to load and it would not.

So, I found a few people who solved this problem by creating a function that would wait for a specific element to load. I decided to go with it, and now my "WaitForElement" function is standard in my local Selenium Template for Visual Studio.

This function looks like this (this is C# code):

private void WaitForElement(string elementName)
{
for (int second = 0; ; second++)
{
if (second >= 120) Assert.Fail("timeout");
try
{
if (selenium.IsElementPresent(elementName)) break;
}
catch (Exception e) { }
Thread.Sleep(1000);
}
}

I call this function the first time I am going to take action after the web app has has to re-load or re-render elements. I tend to pass it the element which I am wanting to interact with first.

So in my test, the steps tend to look like this:
selenium.Click("btn1");
WaitForElement("txtBox1");
selenium.Type("txtBox1", "username");

I have seen some other suggestions from people on stabilizing selenium tests, but would love to hear from anyone else who has worked out a way to make their selenium tests less fail-prone.

Back to blogging ... Selenium and modal dialogs [Confessions of an Agile Tester]

It has been *way* too long since my last blog post, but another post is better late than never, right?

Most recently, I have struggled through automating a heavily AJAX-ey and fairly complex web interface using Selenium.

I have a setup that looks like this:
Visual Studio C# Express
NUnit
Selenium RC
IE Developer Toolbar, for identifying objects on the page
(and now AutoIT)
The occasional use of the Selenium IDE for speed and efficiency's sake.

I was having this particularly stubborn problem with a little dialog that popped up to confirm completion of some action, such as "User profile saved successfully". To this day, I am not 100% certain that I can identify what to call this guy: A pop-up? Alert? Confirmation? IE modal dialog?

Selenium IDE did not record or recognize in any way that this dialog exists. Further, my IE developer toolbar does not recognize this dialog when it's up, either. This all makes identifying it a bit difficult.

In my Selenium script, I tried identifying it as a pop-up, I tried identifying it as an alert, confirmation, and prompt, and none of them worked. My research indicated that this might be an "IE modal dialog" -- which is not supported by Selenium.

As I reached out to the agile-testing Yahoo group for help, I asked the developers how this dialog was being created. They showed me some code that simply called a javascript alert. I am still a little confused about that part, since Selenium RC should capture javascript alerts and close them. (Eventually I figured out that if the alert was called without a page redirect, Selenium behaved as expected, but if it was called as part of a page redirect function, Selenium failed to get it.)

Here is what the code that calls this dialog looks like:

The only difference between that and a similar dialog that Selenium correctly handles are those couple of lines that do the redirect ("parent.SetIframeMainModule"). I am still unclear about why that throws a wrench into my Selenium script, but the bottom line was that my Selenium script was not running!

Chris McMahon responded to my thread on the agile testing list that he had successfully used AutoIT to "hit it with a hammer". After a little bit of searching online, I found a blog post where someone else had used this tool -- in this case, the poster created an executable that looked for windows with certain visible text, and just closes them. This script runs separately from the Selenium test, just constantly polling the windows. It's an ugly hack, but when I tried it, it worked and allowed my Selenium tests to run.

To give proper credit, here is a link to the blog that posted this particular solution:

Get rid of those pesky IE dialogs with AutoIt

I would still love to hear about more graceful solutions ... I understand that Watir has found a way to handle these types of issues, and would love to see Selenium also be able to handle them. Please comment if you have solved a similar problem in a different way!

Why not letting a team self-organize does not work, part 1 [Confessions of an Agile Tester]

What does it mean to let a team self-organize? I am guessing that it varies some from team to team, but I tend to think of concepts like managers trusting the team's opinion on what they need to get things done, empowering them and giving them the freedom to determine their best composition, structure, and timelines.

I have read a lot lately about why that makes sense; how letting the people closest to the work make decisions about it makes sense, and how empowering the teams allows for the best decisions to be made.

So .... why would people say this? Well, again, I am sure you can read all about it from the Ron Jeffries, or Ken Schwaber, or Mary Poppendieck people of the world. Following my precedent, let me be the one to give you a whole bunch of reasons why preventing self-organization *doesn't* work (speaking from experience, of course).

What about team composition? Recently, one member of our team was taken off the team by upper management. That team member had previously been "voted back onto the island" by the team. It bothered me that after our team had chosen for that team member to be included as part of the team, that a member of management would just decide to take them off. What concerned me more was the motivation for doing it (as reported by mgmt): the team has complained that we don't focus enough on the project that has been pounded into our heads as our number 1 priority. This team member focuses mainly on other things, so by removing him from the team, the rest of the team should feel more focused.

The team's response was predictable. They felt like their decision had been undermined by mgmt. The team does not feel empowered to make decisions about what is best for them, and worse, recognizes that the root problem has not been addressed. Removing people from the team does not magically fix the fact that the team is not allowed to focus on the number 1 goal.

A team that is not empowered to make decisions feels disrespected. They feel like they are not trusted and cannot do their best job. These teams do not give their best effort. They stop taking the time to think through everything as thoroughly as they do when they feel passionate, trusted, and empowered. Nobody wins in this situation.

Why not removing impediments does not work [Confessions of an Agile Tester]

What goes wrong when impediments are not removed? Let me put it simply: the team can not succeed! Our team has gone sprint after sprint after sprint saying the *same* things over and over. Here's an example: we use TFS for source control. The box that houses TFS is DOG SLOW .... for those who use the web interface, you click a link, and then go get a cup of coffee ... Every sprint for 9 months, the team has said "we need a better TFS box".

When we didn't get one, it didn't take a lot of time before people resisted using it. And when some people stopped using it (for tracking things like Product Backlog Items and Sprint Backlog Items!!), *that* behavior became an impediment to other team members. Over a few sprints, we eventually dropped the electronic tracking of user story cards altogether!

Back to index cards we went ... but *sigh*. Rather than work toward solving the original problem (faster TFS box!), we started trying to solve the problems that we had with index cards. We have remote employees, and they couldn't see the cards. Worse, the cards lacked detail, and we never have yet solved the problem of several team members not knowing more than just a few words of information about a card.

Our team has spent sprint after sprint trying to work around problems that we have previously identified but not been able to solve, so much so that we are losing sight of our original problems, and are starting to work around problems that some of us have forgotten about.

Why not having an available Product Owner does not work [Confessions of an Agile Tester]

This one seems like a no-brainer, right? Summarizing how I interpret many of the basic ideas behind agile development, a reasonably dedicated, always available product owner is key, right? I feel like agile teams are largely about collaboration and integration of team members, rather than the separation of skills in previous methodologies.

For anyone who for, whatever reason, hasn't read the agile manifesto ... here's a link:
http://agilemanifesto.org/

So, *why* is that the case? *Why* did all of those people get together and publish these ideas, much less advocate for them?

Well, there are plenty of resources out there describing, advocating, and downright fighting for the *why*. I will take the opposite approach and describe why *not*. Here are some of the things that I have seen happen when a team does *not* have a clearly defined, available product owner:

My team has struggled for many many months without a product owner. As we first tried incorporating Scrum into our organization, our first planning meetings were grueling, at best. We had poorly defined ideas of what we should work on. We spent hours asking questions and trying to answer them right there in the planning meetings! We definitely wanted a clear idea of what we were going to do as soon as the planning meeting was over. So we talked through logistics of features and we made decisions.

LET ME TELL YOU .... people in the company who have ideas about what they would like developed DO NOT LIKE when a team makes a decision by themselves! The problem was, they were not around for answering questions. They are located at other geographical locations, and generally do not respond to emails or phone calls in a timely manner. What's a team to do?

So ... we went through a BIG requirements push ... we even hired someone to fill this gap: a dedicated, full time, requirements person. It was partly her job to get the details of desired functionality documented, so that when the team went into planning meetings, they would have the answers to the questions they came up with.

Well, it turns out that "product owners" who are not available to answer questions on the fly, ALSO do not have the time to write requirements. Many planning meetings were entered with still incomplete requirements, or "product owners" promising that they would finish them in a day or two. I guess that maybe having all VPs as your product owners isn't really meant to be the point. Planning meetings became very difficult, as the team started pushing back with saying that they could not estimate incomplete requirements.

Why do I keep saying "they"? Because one of the reasons that having a bunch of product owners is a bad idea is that when they conflict with each other, how can a team solve this problem? We had something like 8 product owners, and each one had an interest in getting *their* features into the development team. Sometimes the team chose what they thought they could do, but were quickly told they were WRONG.

About this time, scrum master #1 had started to burn out, and we had hired a Project Manager (yes, the Gantt chart type). Upper Management decided that the roles should shift, and the previous scrum master would now be a product owner proxy (product owner of product owners, if you will), and the project manager would be the new scrum master (another post soon about why the Gantt chart type has difficulty in scrum master roles!). So ... to solve the "too many product owners" problem, we would funnel everything through one!

Hmm. As it turns out, doing that will not work as long as the product owner proxy does not have any authority to prioritize backlog items. Our PO proxy found himself bringing conflicts to the attention of the other POs, and nobody could make a decision. When he would try to make decisions himself, he even got told "You don't have the authority to make that decision!!"

During this time, upper management decided to create the Sprint Governing Board. The SGB was to meet regularly to decide the priority of story cards. OH WHAT A MESS! In what can only be the most ridiculous planning, the SGB met immediately *after* the planning meeting, and at least 2 sprints in a row, they rearranged the story cards within HOURS of the team spending a day in the planning meeting. This did nothing but make the team angry and tip the scales for a few of the developers into sheer bitterness and outrage.

Most recently, a different person has been given PO responsibility. It's another off-site and unavailable person, previously titled a "product manager". An example of how this goes now ... On Thursday, myself and another team member hit a roadblock; well, a point where we could not proceed without an ok from the PO. We sent her an email (she's not in our office) and told her that we needed help, a decision.

Then ............ we waited. And waited. Thursday night, she sent an email and detailed a whole mess of documentation that we would need to write in order for her to make a decision, and that once that documentation was written, she would be willing to have a conversation about it. To make that frustration even better, a VP followed that up by saying that we *should not* write such documentation, that we should all meet. Then, we waited again. In Monday's stand-up, I said that I was impeded by the fact that I had conflicting instructions from people above my pay grade, and that I needed a decision. After the stand-up, our scrum master scheduled a meeting for Wednesday morning at 10:30.

Do you see the direction this is going ..... ? Even better, do you see the root of the problem? I think I do ... but rather than speculate, I will let you draw your own conclusion. For now, I think I have covered plenty 'o reasons why the lack of a product owner is NOT GOOD.

10,000 ways to not invent a light bulb ... [Confessions of an Agile Tester]

Having grown up in Central Jersey, Thomas Edison's house was the standard field trip fare. I have been thinking a lot lately about that quote from him .... the one about 10,000 ways to *not* invent a light bulb, but only needing one way to get it right.

Hopefully, with each of the 10,000 failures, Thomas (and most of us, with any luck!) learned why the attempt did not work. So, let me see if I can explain the things that I have seen not work, and then more specifically, explain *why* it did not work.

Immediately after our team's stand-up yesterday, we were being told about changes that were coming, set down to us from our "Release Plan Steering Committee" (formerly the Sprint Governing Board). We were told that we were going to try a technique wrt our user story cards that we had not tried before. Some people wondered why we don't handle these things the way(s) that we have all read about, involving user story cards, and breaking the highest priority ones down into tasks in the planning meeting to determine what we thought we could reasonably get done in a sprint.

What I found was the veteran team members started explaining all of the reasons that doing things that way had not worked for us. I started thinking of our original scrum master, and how passionate he had been ... and of Ken Schwaber saying that the average burnout is 18 months. Well, we are on scrum master number 2, and he is trying the same things that this team has already tried, repeatedly ..... So, let me start with the obvious.

Why not having a single, prioritized, and clearly defined product backlog does not work

Here is how this process has (not) worked for us.

When we started trying agile, we had "stuff" that we needed to build. We didn't have user stories, but we did have cards. We tried to go through this process of going into the planning meeting with cards, breaking them down into tasks to complete them and estimating how much we could do. But here was the problem: The team didn't understand enough about the desired functionality to break it down into tasks. As a result, we would spend the better part of a whole day trying to answer the questions we had.

Simple to solve, you say? Just have the Product Owner there, and he/she can answer those questions, and hopefully after a time of two of LOTS of questions, he/she will learn to start thinking through those kind of questions *before* the planning meeting!

Ah, there it was .... that word: "just". Our team has banned that word, among others.

Problem 1 with that statement: To this day, there is not a clear "product owner". There are several, and they conflict with each other, and ALL of their stuff is the Toppermost Toppermost priority. So I will have to write a "Why not having a product owner doesn't work".

At some point, our upper management decided that a *whole day* in planning meetings was too much. So we started having a shorter planning meeting - 1/2 a day. Well, since the lack of defined backlog items problem had not gone away, this just made things worse. We went into even *less* detail in planning, and we were WAY off of our estimates.

Next, they brought in a consultant, a scrum "expert", if you will. He suggested that we stop breaking things down into tasks and that we start writing better user stories. He suggested that we try to break each user story down into some detail on the back of the card, asking the questions of the product owner(s) before the planning meeting. He suggested that in the planning meeting, we just estimate the whole user story.

Okay ..... well, we have been dong that. And, we have struggled with estimating a user story card in story points, and then assigning a number of days to that story card. As one of our team members has said repeatedly, "when you estimate in hours, you are off by hours. when you estimate in days, you are off by days" (I don't know how much I agree with that, but ... it's not a bad concept).

So, the dictate that the team got yesterday from the Steering Committee says that we will no longer put days estimates on story cards. We will only worry about user story points and try to pick off the top points in the planning meeting.

Can you guess how well this will work? I can .. and I think the way to fix the problem, is to fix the problem. We need a single backlog, and it needs to be well-defined and prioritized.

Deprecated ScrumMaster [Confessions of an Agile Tester]

I should be writing this blog ... but I am not. It's got some good stuff about how scrum and agile can be twisted until they are no longer recognizable. I know that feeling oh-so-well .....

Feeds

Feed	RSS	Last fetched	Next fetched after
Abakas	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Canadian Food Inspection Agency - Food Recalls / Allergy Alerts (all)	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Canadian Privacy Law Blog	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Confessions of an Agile Tester	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
coreman's blog	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Geek Etiquette	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Information Technology Dark Side	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Infotropism	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Jeff Fry on Testing	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Lisa Crispin, Agile Testing Practitioner and Coach	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Marlena's Blog	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Matt Heusser's Blog	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Paroxysms of a Geekess	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
PERSONAL FIT MEAL CATERING	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Schneier on Security	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Seebach Exhibit 7	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Testy Testy	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Tyee - Home	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Understanding Nothing	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November
Views from a paintball cynic	XML	08:36, Thursday, 19 November	11:36, Thursday, 19 November