Parallel processing of a list of items using a dynamic number of threads

by Heathesh 20. June 2011 01:46

While working on a project I came across an interesting problem. I had a list of items to process and I wanted to run a certain number of those items at the same time and let the number of items running at the same time be set in a settings file. To illustrate how I did this, let's start of with the items I was going to process. In this case let's say I had a class called "Queue" and each "Queue" item was something I need to do some work with.

    /// <summary>
    /// Queue entity
    /// </summary>
    public class Queue
    {
        /// <summary>
        /// Gets or sets the id
        /// </summary>
        public int Id { get; set; }

        /// <summary>
        /// Gets or sets the name
        /// </summary>
        public string Name { get; set; }
    }

Now in my application, I have a list of these Queue items and I want to be able to process more than one at a time, and I want to be able to set how many items I'm processing at a time in a settings file. So I added a setting file called "QueueSettings.settings" and added a setting called NumberOfThreads which I set to "3".

Now let's pretend my List of queued items to process looks like this:

            List<Queue> queuedItems = new List<Queue>();
           
            queuedItems.Add(new Queue { Id = 1, Name = "Apple" });
            queuedItems.Add(new Queue { Id = 2, Name = "Banana" });
            queuedItems.Add(new Queue { Id = 3, Name = "Orange" });
            queuedItems.Add(new Queue { Id = 4, Name = "Pear" });
            queuedItems.Add(new Queue { Id = 5, Name = "Peach" });

And for simplicity sake, the method that actually processes the Queue item looks like this:

        /// <summary>
        /// Process the queued item
        /// </summary>
        /// <param name="queue"></param>
        private static void processQueuedItem(Queue queue)
        {
            Console.WriteLine("Processed: {0} - {1}", queue.Id, queue.Name);
        }

Now we need to create a method that will call the above method as we require. So we first add the following namespace to our application:

        using System.Threading.Tasks;

Then we create the method that does the actual processing:

        /// <summary>
        /// Processes the queued items
        /// </summary>
        /// <param name="queuedItems"></param>
        /// <param name="threadsToUse"></param>
        private static void processQueuedItems(List<Queue> queuedItems, int threadsToUse)
        {
            //calculate how many times we need to loop, by determining how many times we'll do a full set of
            //items and adding one for the remainder of items. For example, if you have 5 items and you're threading
            //3 at a time, you need to run twice, once to do the first 3, then once to finish off the last 2
            int timesToLoop = ((queuedItems.Count % threadsToUse) > 0 ? 1 : 0) + queuedItems.Count / threadsToUse;

            //set a current record number which we will use to make sure we don't process the same item
            //more than once
            int currentRecordNumber = 0;

            //loop through the amount of times we need to loop
            for (int i = 0; i < timesToLoop; i++)
            {
                if ((currentRecordNumber + threadsToUse) > queuedItems.Count)
                {
                    //this means we're doing the remainder of items
                    int lastThreadsToUse = (queuedItems.Count % threadsToUse);
                    Console.WriteLine("Processing {0} items", lastThreadsToUse);

                    //run in parallel the number of remainder items by calling the process queued item method
                    Parallel.For(0, lastThreadsToUse, _ => processQueuedItem(queuedItems[currentRecordNumber + _]));
                }
                else
                {
                    //this means we're doing a full set of items
                    Console.WriteLine("Processing {0} items", threadsToUse);

                    //run in parallel the number of threads we've been passed by calling the process queued item method
                    Parallel.For(0, threadsToUse, _ => processQueuedItem(queuedItems[currentRecordNumber + _]));
                }

                //increment the current record number by the number of threads we've processed
                currentRecordNumber += threadsToUse;
            }
        }


I'm using a Console application to test this, hence all the Console.WriteLine calls. The method should be self explanatory, it determines how many times it should run and then runs the relevant number of threads each time it is required until it's done. But just for complete disclosure sake, here's the "Main" method of my console app to show you how I tested this:

        /// <summary>
        /// Main application thread
        /// </summary>
        /// <param name="args"></param>
        static void Main(string[] args)
        {
            //create a dummy list of queue items
            List<Queue> queuedItems = new List<Queue>();

            queuedItems.Add(new Queue { Id = 1, Name = "Apple" });
            queuedItems.Add(new Queue { Id = 2, Name = "Banana" });
            queuedItems.Add(new Queue { Id = 3, Name = "Orange" });
            queuedItems.Add(new Queue { Id = 4, Name = "Pear" });
            queuedItems.Add(new Queue { Id = 5, Name = "Peach" });

            //get the number of threads we want to run in parallel from the setting file we
            //created before
            int threadsToUse = QueueSettings.Default.NumberOfThreads;

            //call the method to do the threading etc.
            processQueuedItems(queuedItems, threadsToUse);
        }


Happy Threading!

Tags: , , ,

Development | .Net | Visual Studio 2010 | VS2010

A step by step guide to consuming a web service

by Heathesh 31. May 2011 00:35

I was recently asked for some help from a web developer on how to consume a service, specifically a web service that returns weather information. I thought this would be the perfect opportunity for a new blog post, seeing as I haven't posted in quite a while.

Using Visual Studio 2010, I create a new "Project", specifically an ASP.Net Web Application. I called it "WeatherServiceWebApplication". Next I want to add a reference to my web service. To do this:

1. Right click on "References" and select "Add Service Reference".
2. When the dialog pop's select, select the "Advanced" button on the bottom left hand side of the dialog.
3. Next select the "Add Web Reference" button on the bottom left hand side of the next dialog.
4. Paste the URL of the web service you wish to connect to in the "URL" textbox of the next dialog, in this case I'm connecting to http://iservice.co.za/WeatherInfo.asmx. Then hit enter.
5. I like to change my "Web Reference Name" which you'll find towards the bottom right of the dialog, I changed this to "iServiceWeather".
6. Then click the "Add Reference" button.

Once you've done that, you'll notice a new folder in your solution called "Web References" with the relevant web service reference within it. To do a simple implementation of this, I'm going to make it load some stuff in the default.aspx page.

To begin with, I can see from visiting http://iservice.co.za/WeatherInfo.asmx that there are two methods available on this web service. I'm going to use the GetSACityWeather call to get the weather for Johannesburg. By clicking on the "GetSACityWeather" method name on the previously mentioned URL, I get to see the method definition etc. I specifically want to know what this method call returns, and on this page (http://iservice.co.za/WeatherInfo.asmx?op=GetSACityWeather) I can see the following:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Length: length

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetSACityWeatherResponse xmlns="http://iservice.co.za/">
      <GetSACityWeatherResult>
        <City>string</City>
        <ForecastDate>dateTime</ForecastDate>
        <CurrentCondition>
          <Condition>string</Condition>
          <FahrenheitTemparature>double</FahrenheitTemparature>
          <CelciusTemparature>double</CelciusTemparature>
          <Humidity>string</Humidity>
          <WeatherIconUrl>string</WeatherIconUrl>
          <WindCondition>string</WindCondition>
        </CurrentCondition>
        <ForecastConditions>
          <ForecastCondition>
            <DayOfWeek>Sunday or Monday or Tuesday or Wednesday or Thursday or Friday or Saturday</DayOfWeek>
            <FahrenheitLowTemp>double</FahrenheitLowTemp>
            <FahrenheitHighTemp>double</FahrenheitHighTemp>
            <CelciusLowTemp>double</CelciusLowTemp>
            <CelciusHighTemp>double</CelciusHighTemp>
            <Condition>string</Condition>
            <WeatherIconUrl>string</WeatherIconUrl>
          </ForecastCondition>
          <ForecastCondition>
            <DayOfWeek>Sunday or Monday or Tuesday or Wednesday or Thursday or Friday or Saturday</DayOfWeek>
            <FahrenheitLowTemp>double</FahrenheitLowTemp>
            <FahrenheitHighTemp>double</FahrenheitHighTemp>
            <CelciusLowTemp>double</CelciusLowTemp>
            <CelciusHighTemp>double</CelciusHighTemp>
            <Condition>string</Condition>
            <WeatherIconUrl>string</WeatherIconUrl>
          </ForecastCondition>
        </ForecastConditions>
      </GetSACityWeatherResult>
    </GetSACityWeatherResponse>
  </soap:Body>
</soap:Envelope>


This tells me the response from the service call returns the object specified above, with all the relevant properties etc. available in the XML. So I want to add a few Literals to my default.aspx that I'll populate using this data. So I added the following:

    <p>
        <strong>City:</strong><asp:Literal ID="LiteralCity" runat="server" /></p>
    <p>
        <strong>ForecastDate:</strong><asp:Literal ID="LiteralForecastDate" runat="server" /></p>
    <p>
        <strong>CelciusTemparature:</strong><asp:Literal ID="LiteralCelciusTemparature" runat="server" /></p>



Then I added the following using to the code behind page of my default.aspx:

//WeatherServiceWebApplication is the name of my application and iServiceWeather is the name I gave the service
using WeatherServiceWebApplication.iServiceWeather;

Next, I added properties in my code behind for the Literals I've created on the form:

        /// <summary>
        /// Gets or sets the display city
        /// </summary>
        private string _displayCity
        {
            get { return LiteralCity.Text; }
            set { LiteralCity.Text = value; }
        }

        /// <summary>
        /// Gets or sets the display forecast date
        /// </summary>
        private string _displayForecastDate
        {
            get { return LiteralForecastDate.Text; }
            set { LiteralForecastDate.Text = value; }
        }

        /// <summary>
        /// Gets or sets the display celcius temparature
        /// </summary>
        private string _displayCelciusTemparature
        {
            get { return LiteralCelciusTemparature.Text; }
            set { LiteralCelciusTemparature.Text = value; }
        }


These I put above my Page_Load method, which I then change to:

        /// <summary>
        /// Handles the page load event
        /// </summary>
        /// <param name="sender"></param>
        /// <param name="e"></param>
        protected void Page_Load(object sender, EventArgs e)
        {
            //the iServicecozaWeatherInfo class is created by Visual Studio when it generates the web service reference
            using (iServicecozaWeatherInfo service = new iServicecozaWeatherInfo())
            {
                Weather weather = service.GetSACityWeather(Region.Gauteng, "Johannesburg");
                _displayCity = weather.City;
                _displayForecastDate = weather.ForecastDate.ToString("dd MMMMM yyyy");
                _displayCelciusTemparature = Convert.ToString(weather.CurrentCondition.CelciusTemparature);
            }
        }


That's it. When I run my little web application it will show the following on my web form:

    City:Johannesburg, GP

    ForecastDate:31 May 2011

    CelciusTemparature:9


Happy Servicing!

Tags:

Development | .Net | Visual Studio 2010 | Web Services



Powered by BlogEngine.NET 1.5.0.7 (with enhancements by Heathesh)
Theme by Mads Kristensen (with tweeks by Heathesh)

Certifications

Microsoft Certified Professional

Microsoft Certified Technology Specialist

Answer Questions

 

Tag cloud

Calendar

<<  July 2014  >>
MoTuWeThFrSaSu
30123456
78910111213
14151617181920
21222324252627
28293031123
45678910

View posts in large calendar

http://heathesh.com