[PATCH 1/1] SCons: Updating jobs Command Line Implimentation


Colin Pattison <cpattison@...>
 

Hey

 

This is a small tidy up for the SCons multithreading implementation. The native SCons –j option was essentially being overloaded, which isn’t really the clearest solution.

 

This patch adds a “jobs” argument to the standard command line parser, along with suitable help text. This accepts 1…n, and will calculate the number of available cores if “n” is passed. Since jobs is in the command line variables, it will be saved to the omama.conf file and so be set for all subsequent rebuilds.

 

C


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Damian Maguire <DMaguire@...>
 

Hey Colin, 

Just been taking a quick look at this, and had a few thoughts about the implementation. First of all, the behavior before the change was to utilize as many cores as were available on the build machine, so I don't really think we should change that (this change defaults it to 1). Secondly, I don't like the new 'opt.Add' hanging out by itself – I think it would be better to just include it in the list along with the rest of the general options – the behavior remains identical, and I think it's less confusing/clearer for other users looking at the file.

I've updated your patch with my suggestions, so let me know what you think.

Cheers, 

Damian

---

From 0c0215e7facd6e3222d19010f845acfbfd8fb2b2 Mon Sep 17 00:00:00 2001
From: Colin Pattison <cpattison@...>
Date: Mon, 10 Feb 2014 11:45:39 +0000
Subject: [PATCH] Updating jobs Command Line Implimentation

Previous implementation utilised the scons built in parameter. Adding a new
argument called jobs, which will accepts 1...n as the number of jobs to spawn
where n is all available cores. This will be saved in the conf file and so
will be set for all subsequent rebuilds
---
 SConstruct                           |   10 ++++------
 site_scons/community/command_line.py |    1 +
 site_scons/custom_utils.py           |    3 ++-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/SConstruct b/SConstruct
index 6d96cce..0b1ebc8 100644
--- a/SConstruct
+++ b/SConstruct
@@ -4,7 +4,6 @@ import posixpath,string
 from custom_utils import *
 from SCons.Script import *
 import subprocess,time,shlex
-import multiprocessing
 
 from config import ParseIni
 
@@ -36,11 +35,10 @@ opts.Save('omama.conf', env)
 
 Help(opts.GenerateHelpText(env, sort=True))
 
-try:
-    SetOption('num_jobs', multiprocessing.cpu_count())
-    print "running with -j", GetOption('num_jobs')
-except:
-    pass
+if env['jobs'] == 'n':
+    env['jobs'] = host['cpus']
+
+SetOption('num_jobs', env['jobs'])
 
 if (env['prefix'] == '#wombat_products_%s' % (VERSIONS['mama']['releaseString'])
     or  env['prefix'] == '#openmama_install_%s' % (VERSIONS['mama']['releaseString'])):
diff --git a/site_scons/community/command_line.py b/site_scons/community/command_line.py
index 7614ec8..713f267 100644
--- a/site_scons/community/command_line.py
+++ b/site_scons/community/command_line.py
@@ -27,6 +27,7 @@ def get_command_line_opts( host, products, VERSIONS ):
        BoolVariable('entitled','Whether the build is entitled or unentitled',False),
        PathVariable('gtest_home','Path to Google Test home',None, PathVariable.PathIsDir),
        ListVariable('middleware','Middleware(s) to be compiled in', 'avis', names = ['avis', 'qpid'] ),
+       ('jobs', 'Number of scons threads to spawn, if n is passed the number of availabe cores is calculated and used', 'n'),
 
     )
 
diff --git a/site_scons/custom_utils.py b/site_scons/custom_utils.py
index 4d3152e..49ae616 100644
--- a/site_scons/custom_utils.py
+++ b/site_scons/custom_utils.py
@@ -5,7 +5,7 @@ import shutil,ConfigParser
 
 import os
 import sys
-import platform
+import platform,multiprocessing
 
 # workaround for http://bugs.python.org/issue7860
 # platform.machine on 32b python returns 32 on 64b machine
@@ -40,6 +40,7 @@ def getHostInfo():
     host['arch'] = machine
     host['os'] = system
     host['hostname'] = hostname
+    host['cpus'] = multiprocessing.cpu_count()
     return host
 
 def createDir( dir ):
-- 
1.7.1



Damian Maguire – Senior R&D and OpenMAMA Specialist
IntercontinentalExchange | NYSE Technologies
24-26 Adelaide Exchange | Belfast, BT2 8GD
Tel: +44 2890 822 282 (ext: 452161) | Mob: +44 7540 204 077

From: Colin Pattison <cpattison@...>
Date: Monday, February 10, 2014 11:55 AM
To: "openmama-dev@..." <openmama-dev@...>
Subject: [Openmama-dev] [PATCH 1/1] SCons: Updating jobs Command Line Implimentation

Hey

 

This is a small tidy up for the SCons multithreading implementation. The native SCons –j option was essentially being overloaded, which isn’t really the clearest solution.

 

This patch adds a “jobs” argument to the standard command line parser, along with suitable help text. This accepts 1…n, and will calculate the number of available cores if “n” is passed. Since jobs is in the command line variables, it will be saved to the omama.conf file and so be set for all subsequent rebuilds.

 

C


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Damian Maguire <DMaguire@...>
 

Hey Colin, 

I've raised this into our Bugzilla, just to make it a little easier to track, and attached a patch with my updated changes. Can you take a look at the BZ ticket and stick any comments you have on it when you get a chance, and we can progress the ticket from there?

Cheers, 

D

From: Damian Maguire <dmaguire@...>
Date: Friday, February 21, 2014 9:53 AM
To: Colin Pattison <cpattison@...>, "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH 1/1] SCons: Updating jobs Command Line Implimentation

Hey Colin, 

Just been taking a quick look at this, and had a few thoughts about the implementation. First of all, the behavior before the change was to utilize as many cores as were available on the build machine, so I don't really think we should change that (this change defaults it to 1). Secondly, I don't like the new 'opt.Add' hanging out by itself – I think it would be better to just include it in the list along with the rest of the general options – the behavior remains identical, and I think it's less confusing/clearer for other users looking at the file.

I've updated your patch with my suggestions, so let me know what you think.

Cheers, 

Damian

---

From 0c0215e7facd6e3222d19010f845acfbfd8fb2b2 Mon Sep 17 00:00:00 2001
From: Colin Pattison <cpattison@...>
Date: Mon, 10 Feb 2014 11:45:39 +0000
Subject: [PATCH] Updating jobs Command Line Implimentation

Previous implementation utilised the scons built in parameter. Adding a new
argument called jobs, which will accepts 1...n as the number of jobs to spawn
where n is all available cores. This will be saved in the conf file and so
will be set for all subsequent rebuilds
---
 SConstruct                           |   10 ++++------
 site_scons/community/command_line.py |    1 +
 site_scons/custom_utils.py           |    3 ++-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/SConstruct b/SConstruct
index 6d96cce..0b1ebc8 100644
--- a/SConstruct
+++ b/SConstruct
@@ -4,7 +4,6 @@ import posixpath,string
 from custom_utils import *
 from SCons.Script import *
 import subprocess,time,shlex
-import multiprocessing
 
 from config import ParseIni
 
@@ -36,11 +35,10 @@ opts.Save('omama.conf', env)
 
 Help(opts.GenerateHelpText(env, sort=True))
 
-try:
-    SetOption('num_jobs', multiprocessing.cpu_count())
-    print "running with -j", GetOption('num_jobs')
-except:
-    pass
+if env['jobs'] == 'n':
+    env['jobs'] = host['cpus']
+
+SetOption('num_jobs', env['jobs'])
 
 if (env['prefix'] == '#wombat_products_%s' % (VERSIONS['mama']['releaseString'])
     or  env['prefix'] == '#openmama_install_%s' % (VERSIONS['mama']['releaseString'])):
diff --git a/site_scons/community/command_line.py b/site_scons/community/command_line.py
index 7614ec8..713f267 100644
--- a/site_scons/community/command_line.py
+++ b/site_scons/community/command_line.py
@@ -27,6 +27,7 @@ def get_command_line_opts( host, products, VERSIONS ):
        BoolVariable('entitled','Whether the build is entitled or unentitled',False),
        PathVariable('gtest_home','Path to Google Test home',None, PathVariable.PathIsDir),
        ListVariable('middleware','Middleware(s) to be compiled in', 'avis', names = ['avis', 'qpid'] ),
+       ('jobs', 'Number of scons threads to spawn, if n is passed the number of availabe cores is calculated and used', 'n'),
 
     )
 
diff --git a/site_scons/custom_utils.py b/site_scons/custom_utils.py
index 4d3152e..49ae616 100644
--- a/site_scons/custom_utils.py
+++ b/site_scons/custom_utils.py
@@ -5,7 +5,7 @@ import shutil,ConfigParser
 
 import os
 import sys
-import platform
+import platform,multiprocessing
 
 # platform.machine on 32b python returns 32 on 64b machine
@@ -40,6 +40,7 @@ def getHostInfo():
     host['arch'] = machine
     host['os'] = system
     host['hostname'] = hostname
+    host['cpus'] = multiprocessing.cpu_count()
     return host
 
 def createDir( dir ):
-- 
1.7.1



Damian Maguire – Senior R&D and OpenMAMA Specialist
IntercontinentalExchange | NYSE Technologies
24-26 Adelaide Exchange | Belfast, BT2 8GD
Tel: +44 2890 822 282 (ext: 452161) | Mob: +44 7540 204 077

From: Colin Pattison <cpattison@...>
Date: Monday, February 10, 2014 11:55 AM
To: "openmama-dev@..." <openmama-dev@...>
Subject: [Openmama-dev] [PATCH 1/1] SCons: Updating jobs Command Line Implimentation

Hey

 

This is a small tidy up for the SCons multithreading implementation. The native SCons –j option was essentially being overloaded, which isn’t really the clearest solution.

 

This patch adds a “jobs” argument to the standard command line parser, along with suitable help text. This accepts 1…n, and will calculate the number of available cores if “n” is passed. Since jobs is in the command line variables, it will be saved to the omama.conf file and so be set for all subsequent rebuilds.

 

C


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Damian Maguire <DMaguire@...>
 

That would definitely make more sense if I included the BZ ticket number:



D

From: Damian Maguire <dmaguire@...>
Date: Tuesday, March 18, 2014 2:24 PM
To: Colin Pattison <cpattison@...>, "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH 1/1] SCons: Updating jobs Command Line Implimentation

Hey Colin, 

I've raised this into our Bugzilla, just to make it a little easier to track, and attached a patch with my updated changes. Can you take a look at the BZ ticket and stick any comments you have on it when you get a chance, and we can progress the ticket from there?

Cheers, 

D

From: Damian Maguire <dmaguire@...>
Date: Friday, February 21, 2014 9:53 AM
To: Colin Pattison <cpattison@...>, "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH 1/1] SCons: Updating jobs Command Line Implimentation

Hey Colin, 

Just been taking a quick look at this, and had a few thoughts about the implementation. First of all, the behavior before the change was to utilize as many cores as were available on the build machine, so I don't really think we should change that (this change defaults it to 1). Secondly, I don't like the new 'opt.Add' hanging out by itself – I think it would be better to just include it in the list along with the rest of the general options – the behavior remains identical, and I think it's less confusing/clearer for other users looking at the file.

I've updated your patch with my suggestions, so let me know what you think.

Cheers, 

Damian

---

From 0c0215e7facd6e3222d19010f845acfbfd8fb2b2 Mon Sep 17 00:00:00 2001
From: Colin Pattison <cpattison@...>
Date: Mon, 10 Feb 2014 11:45:39 +0000
Subject: [PATCH] Updating jobs Command Line Implimentation

Previous implementation utilised the scons built in parameter. Adding a new
argument called jobs, which will accepts 1...n as the number of jobs to spawn
where n is all available cores. This will be saved in the conf file and so
will be set for all subsequent rebuilds
---
 SConstruct                           |   10 ++++------
 site_scons/community/command_line.py |    1 +
 site_scons/custom_utils.py           |    3 ++-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/SConstruct b/SConstruct
index 6d96cce..0b1ebc8 100644
--- a/SConstruct
+++ b/SConstruct
@@ -4,7 +4,6 @@ import posixpath,string
 from custom_utils import *
 from SCons.Script import *
 import subprocess,time,shlex
-import multiprocessing
 
 from config import ParseIni
 
@@ -36,11 +35,10 @@ opts.Save('omama.conf', env)
 
 Help(opts.GenerateHelpText(env, sort=True))
 
-try:
-    SetOption('num_jobs', multiprocessing.cpu_count())
-    print "running with -j", GetOption('num_jobs')
-except:
-    pass
+if env['jobs'] == 'n':
+    env['jobs'] = host['cpus']
+
+SetOption('num_jobs', env['jobs'])
 
 if (env['prefix'] == '#wombat_products_%s' % (VERSIONS['mama']['releaseString'])
     or  env['prefix'] == '#openmama_install_%s' % (VERSIONS['mama']['releaseString'])):
diff --git a/site_scons/community/command_line.py b/site_scons/community/command_line.py
index 7614ec8..713f267 100644
--- a/site_scons/community/command_line.py
+++ b/site_scons/community/command_line.py
@@ -27,6 +27,7 @@ def get_command_line_opts( host, products, VERSIONS ):
        BoolVariable('entitled','Whether the build is entitled or unentitled',False),
        PathVariable('gtest_home','Path to Google Test home',None, PathVariable.PathIsDir),
        ListVariable('middleware','Middleware(s) to be compiled in', 'avis', names = ['avis', 'qpid'] ),
+       ('jobs', 'Number of scons threads to spawn, if n is passed the number of availabe cores is calculated and used', 'n'),
 
     )
 
diff --git a/site_scons/custom_utils.py b/site_scons/custom_utils.py
index 4d3152e..49ae616 100644
--- a/site_scons/custom_utils.py
+++ b/site_scons/custom_utils.py
@@ -5,7 +5,7 @@ import shutil,ConfigParser
 
 import os
 import sys
-import platform
+import platform,multiprocessing
 
 # platform.machine on 32b python returns 32 on 64b machine
@@ -40,6 +40,7 @@ def getHostInfo():
     host['arch'] = machine
     host['os'] = system
     host['hostname'] = hostname
+    host['cpus'] = multiprocessing.cpu_count()
     return host
 
 def createDir( dir ):
-- 
1.7.1



Damian Maguire – Senior R&D and OpenMAMA Specialist
IntercontinentalExchange | NYSE Technologies
24-26 Adelaide Exchange | Belfast, BT2 8GD
Tel: +44 2890 822 282 (ext: 452161) | Mob: +44 7540 204 077

From: Colin Pattison <cpattison@...>
Date: Monday, February 10, 2014 11:55 AM
To: "openmama-dev@..." <openmama-dev@...>
Subject: [Openmama-dev] [PATCH 1/1] SCons: Updating jobs Command Line Implimentation

Hey

 

This is a small tidy up for the SCons multithreading implementation. The native SCons –j option was essentially being overloaded, which isn’t really the clearest solution.

 

This patch adds a “jobs” argument to the standard command line parser, along with suitable help text. This accepts 1…n, and will calculate the number of available cores if “n” is passed. Since jobs is in the command line variables, it will be saved to the omama.conf file and so be set for all subsequent rebuilds.

 

C


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.