This project is read-only.

Patch for running a specific scenario

Jun 1, 2010 at 2:32 AM

As part of a project that I'm working on I wanted to write a NUnit test class wrapper around a Narrator Story / Feature to make it easier to execute features and scenarios from within my IDE (SharpDevelop) or using other thirdparty tools that didn't natively support NBehave. However in order to accomplish this I had to make a few changes to NBehave to allow it to execute a specified scenario which is identified by it's name and the name of it's encompasing feature (should it exist). I would like to submit this patch should you want to incorrporate the changes. I however couldn't find a way to accomplish this on Google Code and thought I would instead just post a message on you discussion board asking if you would be interested in the patch and if so how would you like me to send it too you. I'm also more than happy to include the NUnit Style wrapper as well (see bellow for an example).

Calculator Example:

[TestFixture]
public class AddNumbersRunner : TestRunnerBase
{
       // The name of the text file embedded in this assembly
	public override string StoryResourceName {
		get {
			return "AddNumbers.feature";
		}
	}
	
	public override string FeatureName {
		get {
			return "add, subtract, divide and multiply";
		}
	}
	
	[Test]
	public void AddNumbers() 
	{
		ExecuteScenario("Add Numbers");
	}
}

Note: The TestRunnerBase handles interfacing with NBehave.

Jun 1, 2010 at 11:20 AM

Please create an issue on the Google Code site and attach your patch onto that.  I'm a little unclear as to what you are achieving with your change, but it should become once I see the changes you would like to make.  :-)

Kind Regards,
John Rayner

PS. Please ensure that you include in your patch a unit test that fails against current functionality, but passes against patched code.

Jun 3, 2010 at 1:42 AM

I have created a new issue on Google Code (Issue 60).

Sorry about my description earlier I got two different ideas a little mixed up.


Idea 1: Allow TextRunner to execute a specific scenario (This is what the patch is for)

The idea here is that the TextRunner, well any runner really, should be able to execute a specific scenario identified by the name of the scenario and it's encompassing feature.

Please see the issue and patch mentioned above for more information.

Idea 2: Make it easy to create a NUnit TestFixture that wraps a NBehave story.

The idea here is to be able to create a new NUnit TestFixture, much like the example in my earlier post, that allows the user to quickly execute NBehave tasks using testing tools that support NUnit but don't yet support NBehave. While I consider this to defiantly be a workaround the lack of execution support from within my development environment is defiantly a show stopper for me and this was the quickest way I could get up and running in this regard. However I would not consider this a long term solution but am more than happy to provide the code if you want it. Note that the current implementation depends on the latest NBehave source + the patch for Issue 1.

 

 

Jun 3, 2010 at 1:51 AM

Sorry just realized that I missed your comment about a Unit Test. Anyway I did add a few unit tests but they won't compile without the patch since the TextRunner doesn't include the the RunScenario method which is required. Is this what you mean?

Jun 3, 2010 at 10:57 AM
Edited Jun 3, 2010 at 10:58 AM
A compilation error is indeed is failure! The point is to ensure that there are some unit tests that cover the new functionality included in the patch. I'm afraid I'll only be able to look at your patch in a day or two, but I'll let you know when it's committed.

I have to say that I like your "Idea 1" quite a bit. I'm currently investigating Resharper support for scenario files, and it would be great to be able to execute a specific scenario. Does your patch handle Examples at all?

Cheers, John
Jun 3, 2010 at 2:20 PM

No rush in getting this in :), and yes it does support Examples and anything else NBehave supports, both now and in the future. The new functionality actually uses the same code path as a normal "Run" only it's now been altered slightly to take a few predicates to filter out unwanted features and scenarios. In the case of a normal 'Run' every scenario is executed, however for the 'RunScenario' case only the scenario with the specified name is executed.

Oh I just realized the patch also included an update to NUnit 2.5.5, It's what I'm using for my project, however I can resubmit the patch with-out the updated NUnit if you like (or you can just ignore that part).

Jun 14, 2010 at 10:28 PM
A bit later than promised, but your patch has now been committed. :*) Many thanks for taking the time to submit it. FYI I did not commit the NUnit upgrade.
Apr 12, 2011 at 2:11 PM
Edited Apr 12, 2011 at 2:12 PM

I'd like to ask if there is a best practice to wrap a specific feature/scenario run with f.e. NUnit [Test] method (like cweld tried to do (see the first post from the topic)?

1. I see that a specific scenario can be run via

  FeatureResults RunnerBase.RunScenario(string featureName, string scenarioName)

2. How to trace scenario parameters (singular/tabular/examples)?

3. How to report failures for examples data. Some example records can trigger failures, some can pass. NUnit by default stop on first invalid assertion. Maybe dynamically create tests for each record from examples set?

Any other idea to integrate with NUnit (except fluent approach)?

Apr 13, 2011 at 1:10 AM

I'm afraid that we don't currently offer anything more granular than executing all the scenarios contained within a file, either from the command line or from the Visual Studio plugin.  I'd like to add the ability to execute just a single scenario, or even just a single example, but that will be a future task.  I think that this will also help resolve your third point.  Sorry to be of such limited help.

As a debugging tip, if you have multiple scenarios contained within a single file, then you may find it helpful to add a method decorated with [BeforeScenario].  A breakpoint in this method will then let you skip whole scenarios.

Can I ask why you want to integrate with NUnit?  I'm curious to know what benefit it will bring you.

Cheers,
John 

Apr 13, 2011 at 9:19 AM

NUnit is just example testing framework I tried to integrate NBeahve (and ver. 3 will have parametrized test - that can be helpful for granular scenario integration).

I look for any continuous testing integration solution that can generate human readable testing feedback (f.e. in HTML).

I noticed that /xml option is available for generating XML output in NBehave-Console.exe - /html option (in the box) will be helpful as well.

I see that NBehave project contains many testing framework related sub-projects (in NBehave.Spec folder: mbunit mstest9/10 nunit xunit) but I do not know (yet) how to use/integrate them with story testing - any tips are welcome ;)

BTW I'm comparing NBehave functionality to JBehave (Java BDD framework we've chosen) - following features would be helpful in NBehave:

a. Alias attribute (http://jbehave.org/reference/stable/aliases.html)

b. Ability to load examples data from external sources (http://jbehave.org/reference/stable/parametrised-scenarios.html)

c. Web Runner equivalent (see: http://jbehave.org/reference/web/latest/index.html) for ad-hoc scenario testing

Apr 13, 2011 at 9:41 AM
I've added feature requests for Aliases and External tables to our issues page: http://code.google.com/p/nbehave/issues/list

A web runner is not planned at this time, the emphasis is adding text based scenario features and the toolset around that.

There should be a stylesheet which you can apply to the xml file to get a HTML formatted view of the report.

On Wed, Apr 13, 2011 at 9:19 AM, floresek <notifications@codeplex.com> wrote:

From: floresek

NUnit is just example testing framework I tried to integrate NBeahve (and ver. 3 will have parametrized test - that can be helpful for granular scenario integration).

I look for any continuous testing integration solution that can generate human readable testing feedback (f.e. in HTML).

I noticed that /xml option is available for generating XML output in NBehave-Console.exe - /html option (in the box) will be helpful as well.

I see that NBehave project contains many testing framework related sub-projects (in NBehave.Spec folder: mbunit mstest9/10 nunit xunit) but I do not know (yet) how to use/integrate them with story testing - any tips are welcome ;)

BTW I'm comparing NBehave functionality to JBehave (Java BDD framework we've chosen) - following features would be helpful in NBehave:

a. Alias attribute (http://jbehave.org/reference/stable/aliases.html)

b. Ability to load examples data from external sources (http://jbehave.org/reference/stable/parametrised-scenarios.html)

c. Web Runner equivalent (see: http://jbehave.org/reference/web/latest/index.html) for ad-hoc scenario testing

Read the full discussion online.

To add a post to this discussion, reply to this email (NBehave@discussions.codeplex.com)

To start a new discussion for this project, email NBehave@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Apr 13, 2011 at 10:03 AM

John - thanks for reply ;)

Are there any tips available related to my question about: mbunit mstest9/10 nunit xunit?

Apr 13, 2011 at 10:42 AM

The code in the NBehave.Spec namespace is to support the fluent syntax of writing unit tests.  Currently for executing scenarios, we support the following options:

  • Visual Studio plugin
  • command line
  • MSBuild task
  • NAnt task
  • TestDriven.Net (although this is currently limited to running all scenario files in a project)

It sounds to me as if you want to be able to drive scenario files through NUnit / MbUnit / etc tests.  Now I'm not entirely clear on the benefits of doing this, but the code you will need is largely as follows:

      NBehaveConfiguration.New
                .SetAssemblies(new[] { "MyAssemblyContainingActionStepClasses.dll" })
                .SetEventListener(EventListeners.NullEventListener())
                .SetScenarioFiles(new[] { @"MyScenarioFile.feature" })
                .Run();

Cheers,
John 

PS. I can't take credit for the previous reply you got.  ;-)

Apr 13, 2011 at 10:45 AM
It didn't occur to me we could use the configuration in that way, no idea why since that's how we drive nbehave within nbehaves own unit tests! Silly me :-)

On Wed, Apr 13, 2011 at 10:42 AM, john_rayner <notifications@codeplex.com> wrote:

From: john_rayner

The code in the NBehave.Spec namespace is to support the fluent syntax of writing unit tests. Currently for executing scenarios, we support the following options:

  • Visual Studio plugin
  • command line
  • MSBuild task
  • NAnt task
  • TestDriven.Net (although this is currently limited to running all scenario files in a project)

It sounds to me as if you want to be able to drive scenario files through NUnit / MbUnit / etc tests. Now I'm not entirely clear on the benefits of doing this, but the code you will need is largely as follows:

NBehaveConfiguration.New
.SetAssemblies(new[] { "MyAssemblyContainingActionStepClasses.dll" })
.SetEventListener(EventListeners.NullEventListener())
.SetScenarioFiles(new[] { @"MyScenarioFile.feature" })
.Run();

Cheers,
John

PS. I can't take credit for the previous reply you got. ;-)

Read the full discussion online.

To add a post to this discussion, reply to this email (NBehave@discussions.codeplex.com)

To start a new discussion for this project, email NBehave@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Apr 13, 2011 at 4:01 PM

Benefits of wrapping individual scenario tests with f.e. NUnit tests are that you can integrate NBehave tests with testing framework (used in organization) (where other tests are implemented and integrated: unit/UI/etc.) (why do you support fluent syntax tests so? ;))

Test frameworks can generate consistent reports for all kinds of tests then.

Apr 13, 2011 at 4:23 PM

Yeah, I can see it makes sense if you already have other tests and want to execute all tests from a single command.  OTOH you could always construct a NAnt / MSBuild script which invoked a variety of test frameworks.  <shrug/>

Support for the fluent syntax is actually an interesting point.  Firstly, it's more approachable by developers who are getting into BDD.  It provides an alternative to them jumping straight into executable scenario files.  And secondly, the tooling support for test frameworks (e.g. NUnit) is simply superior to the tools that NBehave currently offers for executable scenarios.  This is something that we are working to address (e.g. the Visual Studio plugin is a big step in the right direction) but there is a lot still needed, some of which you pointed out earlier in the thread.  :-)

Cheers,
John 

Apr 14, 2011 at 12:34 AM

For me the main benefits of having an integrated NBehave \ NUnit setup is indeed the tooling support that already exists for the major Unit Testing frameworks, for example being able to easily execute and debug a single scenario from within the IDE, not to mention the report generation tools that are available for these frameworks. A good example of another Behavior Driven framework that goes this particular root is SpecFlow where they have a Custom Tool for Visual Studio which analyses a scenario file and then generates a test harness for your favorite unit testing framework. That way they can rely on the already existing Tools for executing the scenarios and generating the reports while focusing there efforts on the core framework.

Apr 14, 2011 at 10:13 AM

So far in this thread we've mentioned a number of areas where NBehave can be improved:

  • Execute / debug a single scenario or example from a file from within Visual Studio
  • Trace success and failure of individual examples when running scenarios
  • Alias attribute and external tables [already added into the issue list by Naeem]

But none of these cover test reports ... are you talking about something like the Test Results window within Visual Studio or an HTML report produced on a build server?  And are there any other specific suggestions you could make for improving the NBehave tooling?

Cheers,
John 

Apr 14, 2011 at 4:57 PM
Edited Apr 14, 2011 at 5:00 PM

1. I mentioned earlier about testing framework reporting benefits ;)

2. BTW - I've just tried to attach to NBehave-Console.exe (with option /wd). Although I was be able to attach to the process - VS did not entered into Debug mode.

A timeout before quiting the process (NBehave-Console.exe) is too short (about 2 seconds) - I really had a problem to attach the process before it was terminated (most times I got: "fatal error: timeout while waiting for debugger to attach").

I suggest the timeout can set by options.

I managed Debug by defining "Start external program" field in project properties->Debug tab (and proper parameters as well in "Command line arguments" and "Working directory") or by adding System.Diagnostics.Debugger.Break(); in method decorated with [BeforeScenario]

Apr 14, 2011 at 7:44 PM
Edited Apr 14, 2011 at 7:54 PM

A) is already solved, just add multiple attributes to the same mothod :)

B) Alread in the box, see this

C) Web runner would be nice

Apr 14, 2011 at 7:47 PM
Edited Apr 14, 2011 at 7:55 PM

There is a stylesheet to render the xml output into html, I think it somehow got removed from the zip / installer but you can grab it here. I have used it successfully with Cruiscontrol and TeamCity.

Apr 14, 2011 at 7:53 PM
Edited Apr 14, 2011 at 7:56 PM

2 seconds is way to short, I noticed that myself :)

So If you grab the latest build (here) you will have 15 seconds to attach the debugger + a few small bugfixes compared to the official 0.5 release.

/Morgan

Apr 15, 2011 at 8:17 AM

1. Thanks for xslt link (I spent a while for creating sample xslt - because I could not find any in the source code)

2. Ad B) I know that NBehave supports examples - but I meant ability to get examples' content from an external source (not from the feature file) - best if also simple interface will be available with a method returning examples' content

Apr 15, 2011 at 8:53 AM

Ah, I didnt scroll all the way down :)

The xslt file should also be in the distributed zip if you get the latest download.