Chris Menegay's Weblog

Testing with Team System

Chris Menegay — Tue, 01 Jan 2008 16:03:00 GMT

At Notion, we get a lot of random questions to some of our generic email accounts. Just recently, I've become the person these get routed to. I don't answer them all, it's not our job to be free tech support for the world ;-), but I answer some, specifically the ones that sound like they might be legitimate business opportunities. Anyway - I decided to start posting the questions and answers to my blog, so they might benefit more people. Here's the first - names stripped to protect identity:

Our products are complex and varied – using primarily the Windows platform (We are MS Developers utilizing MS Visual Studio, Server 2003, MS SQL 2005 and XP Pro primarily but not exclusively).

We use .NET & C# with Web and Windows applications over about a dozen apps with several plug ins etc…. (a “suite” if you will)

I’ve been looking at a product we can upgrade to.(we have a previous license with TestComplete) and we also are being pushed by the developers to stick solely with MS products such as Visual Studio/Team which they use – and we (QA) use just the ‘Bug Tracking’ part of it – not being code oriented developers ourselves – but needing to communicate with them on bugs etc..

Our code moves a bit fast – and we’re under the impression that very little automation is going to help us all that much – but we need to confirm this. As well as seeing if any coding or testing directly in MS Visual Studio/Team will give us the benefits of (I believe) what I think we need – which is ‘object oriented’ testing capabilities.

Can you give us your insight on all these matters – and perhaps tell us what your impressions and recommendations would be?

My answer:

What I normally recommend is using Team System to track your defects and manage your testing (test plans, test runs, etc.). This gives you a central repository for your tests and test results that you can use for reporting. Team System lacks a good functional UI tester (GUI automation) – for this, I would continue using TestComplete which integrates nicely with Team System. You can use Team System’s web testing capabilities to test your web applications, though some of the test mechanics can get very “code oriented”. A generally good strategy is having the QA team record the web tests, and have the development team maintain and run them as they modify the code.

You can get some more information from the following videos:

http://vstsrocks.com/files/25/vste-test_new/default.aspx

as well as the complete VSTS Overview (you can just download and fast forward to the testing part) located here:

http://vstsrocks.com/files/10/vsts/default.aspx

Answers to Orcas Team Build and Folder Diff questions from the DNUG meeting

Chris Menegay — Fri, 14 Sep 2007 15:15:00 GMT

Ok, so at the Dallas .NET Users Group last night, some questions were asked about Orcas that I didn't have answers to - I know, what a shock! Sometimes I miss the good old days of VSTS 2005, where all the good questions had been asked and I knew pretty much all the answers before someone even finished asking the question... now I get to be clueless again... I didn't even let people ask Rosario questions, I'd already told them all that I know (didn't take long), and getting answers out of MS on Rosario is a bit dodgy right now. Oh, and for those of you that aren't even using VSTS 2005, goodness, get out of the stone ages! Have you even gotten a wheel yet?!? It won't be long and VSTS 2005 will be OLD V1 stuff.

Anyway, the main question was about Team Build, so I turned to Buck Hodges and got a bit of help. The main question was if I have a stack of builds in the build queue, what version of the code is retrieved when the build finally gets run. Here's what Buck said:

"When the server detects a checkin that affects a build definition for which 'build each check-in' is enabled, it records the changeset number in the build queue. When the build occurs, that changeset is passed to the build (what literally happens is that the build process requests the build request information from the server, and that’s part of the build request information). If you are using CI with 'accumulate check-ins,' it also records the version to build in the build queue, but it’s based on the time the build gets queued."

That sounded a bit like magic, and I don't trust magic with my software process, so I opened up my TeamFoundation.Build.targets file to see how this worked and found the following:

<GetVersion Condition=" '$(GetVersion)'=='' ">$(SourceGetVersion)</GetVersion>

I assume that is the value Buck is saying that gets passed in. I didn't dig any further.

The second question that came up was whether folder diff happens on the server or on the client. That was a simpler question. Buck said: "Folder diff is computed on the client." I dug a little deeper, as that seemed like it wouldn't perform well, so Tan Phan (that's a fun name to say!) explained further: "If for comparing two folders on the server, folder diff downloads the folders/files list and compares the MD5 hashes. File contents are not downloaded unless you decide to actually view the diffs of the contents."

Needless to say, I'm happy with both implementations, it seems well thought out.

Getting code coverage for manual testing (or anything really)

Chris Menegay — Thu, 25 Jan 2007 19:26:00 GMT

This question crops up from time to time, and it was suggested that I just post this to my blog, since it seems it's rare enough that people have a hard time finding it :-)

The only reason I even have it is because an old lab that we had at Notion actually had students implement this. The following describes how to get code coverage on .NET applications without having to actually run "tests" in the VSTS context. You will need either Team Dev or Team Test installed to have the right tools to make this work.

Using the commandline you can instrument ANY .NET assembly (code signing can cause problems). Once that code is instrumtented, you need to actual get the "listener" running to collect the code coverage data this generated. Here's the basic steps:

First, you have to open a VS command prompt:

a. Instrument your assembly by executing:

vsinstr /coverage Assemblyname.dll

b. Start the Code Coverage monitoring process by executing:

vsperfcmd /start:coverage /output:yourfilename.coverage

Run your Application that contains the instrumented code

End the Code Coverage monitoring process by executing:

vsperfcmd /shutdown

You should then have a file with a "yourfilename.coverage" file in the application directory that you can open up in Visual Studio to view the results.

Answers to questions about DB Pro

Chris Menegay — Wed, 20 Dec 2006 17:31:00 GMT

I recently did 9 Team Edition for Database Professional launch events in 9 different cities (in 3 weeks - yikes!). As I told all of the attendees, I would gather all of the unanswered questions, and post the answers to them on my blog. I actually got REALLY lucky - at the last event (in Houston), someone from the product group showed up - Robert (I have no idea what is last name is - but thanks Robert!). Robert knocked out all of these questions for me, so the only reason it has taken so long to post them is because of me.... anyway - here goes:

For Reporting Services, OLAP, SSIS, DTS support, there isn't any better answer than "not at this time" :-)

Can I migrate a VS2003 Database Project up to new one?
--Import the sql scripts, or deploy and reverse engineer

How does it handle a encrypted stored proc?
--you need to have the original (unencrypted) SPROC in source countrol and deploy it as encrypted

If i have multiple DBProjects, will refactor look in them?
--NO

If I have one SPROC that hits another DB, does that build? if not, how do I work around it?
--In local instance, you'll need to create a linked server so that compile works (shows warnings then instead of errors)

For cross DB call, how are those handled during build/compare?
--will still receive a warning, if for example the linked server reference is resolved

What order are post-deployment scripts run?
--in order that they are included

What if I am developing on SQL2000, what needs to be installed on my machine?
--SQL 2005 instance for the project system + Team Data

What happens if I open the solution with a dbproj on a machine with VS not DBPro?
--The solution will load however you won't be able to access / use the dbproj project; same behavior if you didn't have C# installed and the solution contained a C# project

If I have project/solution in SQL Mgmt Studio, can I move that over easily?
-- Recommend importing / reverse engineer the database vs. import each script into the database project

If i create folders in my project will it build correctly?
-- yes; you can add files & folders to the project; files have a build property to determine if they are to be included in the build process

Can i write my own test validation conditions?
-- yes

In the future, will Team Test allow running db unit tests?
-- This is when the user doesn't have team data - Not today as the generated code inherits from our unittest assembly.

Does "skip" on schema compare change depoy script?
-- yes, it would not include any information in the schema compare deploy script for that object

If I use "description" in a table or column in SQL Mgmt Studio - how can I make use of that?
-- if this is the extended properties, they will be imported as part of reverse engineering a database into the db project

Licensing Team Foundation

Chris Menegay — Wed, 08 Feb 2006 00:24:00 GMT

A question that often comes up with our customers that are adopting Team System is "who needs a CAL?" This has been a confusing topic for some time, as it appears the messaging has changed as the product has gone through it's lifecycle. The current licensing whitepaper suggests that anyone that is pulling data from the Team Foundation Server requires a CAL. Many customers have been assuming that reporting against the database would not require a CAL. I have been told that the latest guidance will be updated to reflect the following:

Running reports that are on the project portal will NOT require a CAL.
Creating reports will require a CAL
Running reports from another, custom website, will require a device CAL for that web server
Anyone editing data will require a CAL

Workspaces in TFVC

Chris Menegay — Fri, 03 Feb 2006 00:24:00 GMT

Ok, I just read http://blogs.vertigosoftware.com/teamsystem/archive/2006/01/27/2051.aspx, which was linked to by the Team System News feed (great job Mickey - YOU rock!).

This posting confused me a bit, as it didn't seem really address the whole workspace concept - which is critical to check-outs. So yes, a check-out simply marks a file as writeable with the idea that it has "pending changes". If you immediately check the file in, the IDE will tell you that there are no changes, and nothing happens. So that left me with an empty feeling that I just had to discuss...

So let's talk about what a workspace is, as this is a common point of confusion for people moving from VSS. A workspace is similar to your local "working directory" in VSS, but it has same major differences. First off, a workspace is "owned" by a user. For example:

I was working with a client that had a person using a laptop and had a workspace mapping to a particular directory (for this example, c:\projectfiles). When your workspace is setup using that directory, the Team Foundation Version Control system "knows" what user is using that directory on that machine. The server keeps track of this, as well as every single file that you have in that directory, and which version of those files you have. This is why if you delete a file using windows explorer and then say "Get Latest" it does nothing. The server believes you have the file, so why waste the network traffic sending it to you again - surely you weren't stupid enough to delete it? :-) Anyway, back to the customer problem - they person using the laptop left the company, and they gave the laptop to another developer. That new developer attempted to use the exact same directory (c:\projectfiles) for their own workspace mapping. TFVC didn't like that, as that workspace was "owned" by the original user, and they weren't allowed to use it. To fix this, you have to delete the workspace from the server (not just delete the files locally).

So - workspaces are owned by users and they are managed at the server level. There are some cool things about this, for example, if you edit your workspace mappings (under the File -> Source Control menu), and change one of your local directory mappings, the Version Control client will "magically" move all the files from the original directory to the new mapped directory - after all, the server now knows where they should be. Another interesting note is that workspace are by user and by machine. So if I login to a 2nd computer, I essentially have a new workspace - with new mappings, and with potentially different files in it. If you login to my computer you will have a different workspace than I do on my computer - just don't use the same directory!

Anyway - I hope this provides some insight into workspaces, as many times as I've explained it to clients, I still don't feel that I have a "simple" description...

A VSTS job posting...

Chris Menegay — Fri, 06 Jan 2006 21:17:00 GMT

I've been out of the country and terribly busy since launch, but I've got lots of good nuggets of things I've learned about VSTS over the past couple months that I need to get out of my head and into the blog. But first...

Notion Solutions is looking for another VSTS trainer/consultant, preferably someone in Houston or Chicago. Travel would be required, as well as a strong interest in helping companies adopt VSTS. Just hit the contact me link on my blog if you or any one you know is interested. You don't have to be fluent on VSTS, we'll get you up to speed.

My talk at VSLive Orlando - it was rough...

Chris Menegay — Wed, 12 Oct 2005 19:28:00 GMT

I just did a talk in the ASP.NET track @ VSLive Orlando. It didn't go well. It wasn't a complete disaster, but it was very rocky. The talk was on Unit Testing, Static Code Analysis, Team Build, and the Profiling tool (basically VSTE for Devs with relation to ASP.NET). I was happy with all of my demos prior to my talk and decided not to shut down my laptop for the walk to the presentation room. I did shut down my VPC and disconnected my external disk. I got to the room 30 minutes early to test everything out - and then the trouble began.... I had VPC go nuts on me. My VPC wouldn't repaint at all while I was typing/mousing. For example, when entering my password to login, it wouldn't show the little ****, it looked frozen. It wasn't frozen, and if I resized the windows a few times, it would repaint. Then I would have to type/mouse more and resize, etc. I restarted the VPC and it was doing the same thing. I finally restarted my laptop and that fixed the problem, but I ended up starting 5 minutes late for my talk. There were a couple of things that I had originally planned to "warm up" when I got there early and they weren't warm, so I was a bit flustered and was very random in my delivery. In all, I wasn't very pleased - I rarely get bitten by the "demo demons", but I got smacked around on this one. Pretty much all of my demos worked, but not exactly how I intended, so it through me off.

VSTS MVPs!

Chris Menegay — Mon, 19 Sep 2005 13:09:00 GMT

I am happy to announce that MS now has a new MVP category called "Visual Developer - VSTS". I'm even more happy that I made the cut :-) There are only 5 currently - all existing MVPs that moved their product affiliation (so long VB!). I have been told that more people will be awarded soon, but not in time for the MVP summit at the end of September. Dave McKinstry also made the list, which you can find at the MVP site. Another VSTS MVP is Jean-Luc David, who also blogs on TeamSystemRocks.com, also made the list - very cool!

I'm now in the INETA speaker's bureau - and other ramblings about VSTS

Chris Menegay — Fri, 02 Sep 2005 14:42:00 GMT

I just found that I've been added to the bureau, I'm curious to see how many VSTS talks I do in next year based on that. They sent me a form for my areas of expertise, and I put nothing but VSTS topics (Build, SCC, project management, testing, etc.) ;-)

For those that subscribe to my blog for useful VSTS info, sorry I haven't put much up lately. It's not that I haven't been doing a lot with it, it's just been a bit hectic. We're wrapping up work on level 400 training right now, which we'll have on Beta 3 once that comes out. Some of that will make it into our normal offerings, some of it is really too deep (certain extensibility topics) for the average person to care.

About a month ago we did our first Team System Quickstart for a customer in Houston, and while that was a bit rocky, it went well and they were happy. They are now working in Beta 2 and we will encourage them to move to Beta 3 ASAP once that is out - that way they can take advantage of the migration path to RTM.

The last topic I'd like to throw out is about some of the things we plan on working on once Beta 3 is out. We have uncovered several "opportunities" for improvement in VSTS, and are trying to decide which to tackle first. Obviously we'll base this on need and on how long each piece will take to do. A lot of this, we are planning on rolling up into some sort of resource kit for VSTS that we'll either use as a value-add to our VSTS services engagements, or we'll sell it for some nominal fee. Some of these we have already done work on, others are still ideas. Here's our current list:

Some sort of work item generator/chaining tool auto create dependent work items based on what happens in your process
Outlook integration - MS is releasing this as a starter kit, but the functionality will be very limited from what I've heard
Administration - There are lots of things that are simply a pain in the rear (and in my opinion, they weren't really thought through) when it comes to managing large teams. The templates for manuals tests reside on the local user's hard drive, so if you modify the template, you have a deployment problem. The same thing is also true for Static Code Analysis rules and checkin policy. I think managing user permissions for SharePoint, RS, and TFS would fall under this as well.
An alert manager. The standard alert list is small, moderately useful, and FIXED - you can't add your own. Obviously, there are events you can write code to listen for and build your own rules. For example, what if I want to know when a specific file is checked in, but I don't care about every checkin? What if I want an email when someone overrides a checkin policy?
Reports - there are some reports we want to write, these will likely be straightfoward. Policy violation reports, bugs created by developer, etc.
A little tool to help with SCC. Primarily, a lot of customers want check-out to also do a "get latest". Team System can't do this, we have some ideas on ways to hack this, but haven't tried yet.
Sarbanes-Oxley targeted software process. We're already working on this and plan to have by the time TFS launches. Basically, it's MSF CMMI with extra data, extra reports, and a plugin to automatically turn on the work item policy.

Anyway, if you have any thoughts or other ideas, let me know!

Generic Test Sample

Chris Menegay — Fri, 12 Aug 2005 16:12:00 GMT

Ok, I couldn't find a good sample for creating a generic test that actually returned results, so I created one. It's up for download on TeamSystemRocks. Feel free to take a look at it. I didn't clean it up at all, but it should provide a decent start for people looking to create their own generic tests.

Our last public training class till October.

Chris Menegay — Thu, 11 Aug 2005 20:07:00 GMT

Ok, this is a shameless marketing bit - At the end of this month we're holding our last public training class for Team System until October. The training class is August 29-31 in Dallas and we've dropped the price to $950 from the normal $1750. We want to fill the class up. We are hoping (hoping - no guarantee) to run part of the class in a shared server environment using one TFS for the whole class for the coding and testing labs. One reason we want a full class is to see if one little old laptop (ok, one BIG laptop) can handle that. We'll have local VPC installs to fallback to if the laptop craters. So, if you been considering coming to one of our classes, or have been considering whether or not you want us to do a private class for your company, this is a fairly inexpensive way to get a feel for class, the material and us ;-).

Also, I'll be posting soon a list of projects that our team is considering to extend the featureset of VSTS, I'd love to get feedback from as many people as possible. That will allow us to better prioritize the list. So hopefully, you guys have given some thought to what you think some of the lacking features are and will let us know!

Visual Studio 2005 Support for .NET 1.0/1.1

Chris Menegay — Thu, 04 Aug 2005 00:24:00 GMT

Ok, apparently this is "old news", but it was new to me. If you want to adopt Visual Studio 2005 (and potentially Team System), and you want to do work on 1.1 projects - check out Robert McLaws posting about his new tool to allow this. This is great news, though I haven't tried it yet. If you get a chance to use this, let me know. I'm excited because this opens up the door for more people to adopt Team System when it releases without having to target the 2.0 framework. We're going to be testing this scenario so that our Team System consulting and training practice can integrate this in.

What's coming in the July CTP

Chris Menegay — Thu, 28 Jul 2005 12:00:00 GMT

The July CTP is coming out soon and I got news of some of what will be included, as well as things that might be included. There are a few things that I’m looking forward to.

The command-line tool for running builds. – Some people have already written their own, it will be nice to have the official one.
New reporting structure – they have supposedly cleaned up the codenames from the operational datastore, and changed the OLAP structure for reporting. I’ve been ignoring reports until they’ve finished this (why learn stuff that’s going to be removed). So now I’ll finally be able to spend some time learning how write some reports. First up – bugs created by developer ;-)
Updated VSS migration tool – supposedly the new tool is going to have better visibility into what happens.
Admin tools – supposedly for some sort of backup and restore.
Source Control Proxy – unfortunately, I have a client that could probably use this, but I’m suggesting they stick with Beta 2 until either another beta comes out, or RTM. There are supposedly some pretty good gains in overall performance.
Stability & Performance – this build is going to be used for Microsoft’s “dogfood” server. That gives me some confidence that at least the product group thinks it’s a pretty stable build.
Performance boosts for the Office integration pieces – Excel and Project can be awful sluggish at times sync’ing work items. This is supposed to be better in the July CTP.

Things I’m hopeful will be in the July CTP, but not sure… I will have to check once I get it:

HTTP support for Team Foundation Server – while it boggles my mind that this wasn’t there from Day 1 (it’s an architecture change IMO), this is supposed to be support by RTM, and _may_ be in the July CTP. It will be nice not to have to VPN.
Support of ASP.NET unit tests _may_ be working in the July CTP – I haven’t tried this in Beta 2, but supposedly it’s problematic.
Revised Static Analysis checkin policy (aka STAN). We’ve been working on some extensibility stuff and some of the checkin policy things seemed like a hack, I’ve been told that this policy was getting reworked, and hopefully that will help us out.

Overall, there’s some pretty good stuff, and it sounds like quite a few bugs are fixed. I have a feeling this is just going to get me anxious for the next beta. The only downer is that it doesn’t support a single server install L.

Team Foundation running on AD2000

Chris Menegay — Tue, 26 Jul 2005 02:01:00 GMT

Well, we successfully got a client up and running with Team Foundation Server on a network that is running AD2000. Actually, we brought up an AD2003 domain controller on the same box running TFS and got a trust relationship set up. The trust was a pain in the rear and required a call to Product Support. Everything seems to be working fine with a couple of small problems:

MS Project integration doesn’t work – it give an error when launched from VS, and hangs when you try and hit work items when launched from the start menu.
Team Explorer doesn’t seem to like fully qualified names. For example, server.domainname doesn’t work and gives errors, where “server” is fine. For now we’ve put “server” in the HOSTS files until I can get an admin at the client to adjust the DNS.

I hope to get the Project problem figured out and will post the work around if discovered.