pygeoc.utils module

Utility Classes and Functions

@author: Liangjun Zhu

@changlog:

• 12-04-12 jz - origin version.
• 16-07-01 lj - reorganized for pygeoc.
• 17-06-25 lj - check by pylint and reformat by Google style.
• 18-10-31 lj - add type hints according to typing package.

pygeoc.utils.DEFAULT_NODATA = -9999.0

Default NoData value for raster dataset.

pygeoc.utils.DELTA = 1e-06

Delta value to check two approximately equal floats.

class pygeoc.utils.DateClass

Bases: object

Utility function to handle datetime.

static day_of_month(year, month)

Day number of month

static day_of_year(dt)

Day index of year from 1 to 365 or 366

static is_leapyear(year)

Is leap year?

Examples

>>> DateClass.is_leapyear(2000)
True
>>> DateClass.is_leapyear(2010)
False
>>> DateClass.is_leapyear(2004)
True
>>> DateClass.is_leapyear(2018)
False

class pygeoc.utils.FileClass

Bases: object

File IO related

static add_postfix(file_path, postfix)

Add postfix for a full file path.

Examples

>>> FileClass.add_postfix('/home/zhulj/dem.tif', 'filled')
'/home/zhulj/dem_filled.tif'
>>> FileClass.add_postfix('dem.tif', 'filled')
'dem_filled.tif'
>>> FileClass.add_postfix('dem', 'filled')
'dem_filled'

static check_file_exists(filename)

Throw exception if the file not existed

static copy_files(filename, dstfilename)

Copy files with the same name and different suffixes, such as ESRI Shapefile.

static get_core_name_without_suffix(file_path)

Return core file name without suffix.

Examples

>>> FileClass.get_core_name_without_suffix(r'/home/zhulj/1990.01.30/test.01.tif')
'test.01'
>>> FileClass.get_core_name_without_suffix(r'C:\zhulj\igsnrr\lreis.txt')
'lreis'
>>> FileClass.get_core_name_without_suffix(r'C:\zhulj\igsnrr\lreis.txt')
'lreis'
>>> FileClass.get_core_name_without_suffix(r'C:/zhulj/igsnrr/lreis.txt')
'lreis'
>>> FileClass.get_core_name_without_suffix(r'/home/zhulj/dta/taudem/aread8')
'aread8'
>>> FileClass.get_core_name_without_suffix('singlename')
'singlename'
>>> FileClass.get_core_name_without_suffix('singlename.txt')
'singlename'

static get_executable_fullpath(name, dirname=None)

get the full path of a given executable name

static get_file_fullpath(name, dirname=None)

Return full path if available.

static get_filename_by_suffixes(dir_src, suffixes)

get file names with the given suffixes in the given directory

参数:

• dir_src – directory path
• suffixes – wanted suffixes list, the suffix in suffixes can with or without ‘.’

返回:

file names with the given suffixes as list

static get_full_filename_by_suffixes(dir_src, suffixes)

get full file names with the given suffixes in the given directory

参数:

• dir_src – directory path
• suffixes – wanted suffixes

返回:

full file names with the given suffixes as list

static is_dir_exists(dirpath)

Check the existence of folder path.

static is_file_exists(filename)

Check the existence of file path.

static is_up_to_date(outfile, basedatetime)

Return true if outfile exists and is no older than base datetime.

static remove_files(filename)

Delete all files with same root as fileName, i.e. regardless of suffix, such as ESRI shapefile

class pygeoc.utils.MathClass

Bases: object

Basic math related.

static floatequal(a, b)

If float a is equal to float b.

Examples

>>> MathClass.floatequal(1, 1.000001)  # result likes -9.999999999177334e-07
True
>>> MathClass.floatequal(1, 1.0000001)
True
>>> MathClass.floatequal(2, 2.)
True
>>> MathClass.floatequal(1, 1.00001)
False

static isnumerical(x)

Check the input x is numerical or not.

Examples

>>> MathClass.isnumerical('78')
True
>>> MathClass.isnumerical('1.e-5')
True
>>> MathClass.isnumerical(None)
False
>>> MathClass.isnumerical('a1.2')
False
>>> MathClass.isnumerical(['1.2'])
False
>>> MathClass.isnumerical(numpy.float64(1.2))
True

static nashcoef(obsvalues, simvalues, log=False, expon=2)

Calculate Nash-Sutcliffe coefficient(NSE) proposed by Nash and Sutcliffe (1970) and its variants.

The following description is referred by Krause et al. (2005) and Moriasi et al. (2007).

• The range of NSE lies between -inf and 1.0 (prefect fit).
• Since the differences between observed and simulated values are calculated as squared values (expon=2), the larger values in a time series are strongly overestimated whereas lower values are neglected (Legates and McCabe, 1999). For the quantification of runoff prediction, this leads to an overestimation of the model performance during peak flows and an underestimation during low flow conditions.
• Similar to R-square, NSE is not very sensitive to systematic model over- or underestimation especially during low flow periods.
• To reduce the sensitivity of the original NSE to extreme values, the NSE is often calculated with logarithmic values of obseravtion and simulation values, which known as lnE. As a result, the influence of the low flow values is increased in comparison to the flood peaks resulting in an increase in sensitivity of lnE to systematic model over- or underestimation.
• A more general form could be used for the same purpose as lnE, i.e., varying the exponent from 1 to N. With the increase of expon, the sensitivity to high flows will increase and could be used when only the high flows are of interest, e.g., for flood prediction.

参数:

• obsvalues – observation values array
• simvalues – simulation values array
• log – Do logarithmic transformation or not, False by default
• expon – The exponent range from 1 to N, 2 by default

Examples

>>> obs = [2.92, 2.75, 2.01, 1.09, 2.87, 1.43, 1.96,                       4.00, 2.24, 29.28, 5.88, 0.86, 13.21]
>>> sim = [2.90, 2.87, 2.85, 2.83, 3.04, 2.81, 2.85,                       2.78, 2.76, 13.40, 2.70, 2.09, 1.62]
>>> MathClass.nashcoef(obs, sim)  
0.451803966838596...
>>> MathClass.nashcoef(obs, sim, log=True)  
0.2841143016830745...
>>> MathClass.nashcoef(obs, sim, expon=1)  
0.3959646306103376...
>>> MathClass.nashcoef(obs, sim, expon=3)  
0.6122272075952075...
>>> MathClass.nashcoef(obs, sim, expon=14)  
0...
>>> MathClass.nashcoef(obs, sim, expon=0)  
0...

返回:NSE, or raise exception

static pbias(obsvalues, simvalues)

Calculate PBIAS, or percent model bias.

参数:

• obsvalues – observe values array
• simvalues – simulate values array

Examples

>>> obs = [2.92, 2.75, 2.01, 1.09, 2.87, 1.43, 1.96,                       4.00, 2.24, 29.28, 5.88, 0.86, 13.21]
>>> sim = [2.90, 2.87, 2.85, 2.83, 3.04, 2.81, 2.85,                       2.78, 2.76, 13.40, 2.70, 2.09, 1.62]
>>> MathClass.pbias(obs, sim)  
35.46099290780142...

返回:PBIAS value (percentage), or raise exception

static rmse(obsvalues, simvalues)

Calculate RMSE.

参数:

• obsvalues – observe values array
• simvalues – simulate values array

Examples

>>> obs = [2.92, 2.75, 2.01, 1.09, 2.87, 1.43, 1.96,                       4.00, 2.24, 29.28, 5.88, 0.86, 13.21]
>>> sim = [2.90, 2.87, 2.85, 2.83, 3.04, 2.81, 2.85,                       2.78, 2.76, 13.40, 2.70, 2.09, 1.62]
>>> MathClass.rmse(obs, sim)  
5.590926715533082...

返回:RMSE value

static rsquare(obsvalues, simvalues)

Calculate Coefficient of determination.

Same as the square of the Pearson correlation coefficient (r), and, the same as the built-in Excel function RSQ(). Programmed according to equation (1) in

Legates, D.R. and G.J. McCabe, 1999. Evaluating the use of “goodness of fit” measures in hydrologic and hydroclimatic model variation. Water Resources Research 35:233-241.

参数:

• obsvalues – observe values array
• simvalues – simulate values array

Examples

>>> obs = [2.92, 2.75, 2.01, 1.09, 2.87, 1.43, 1.96,                       4.00, 2.24, 29.28, 5.88, 0.86, 13.21]
>>> sim = [2.90, 2.87, 2.85, 2.83, 3.04, 2.81, 2.85,                       2.78, 2.76, 13.40, 2.70, 2.09, 1.62]
>>> MathClass.rsquare(obs, sim)  
0.7528851650345053...

返回:R-square value, or raise exception

static rsr(obsvalues, simvalues)

Calculate RSR (RMSE-to-SD Ratio).

Programmed according to equation (3) in Moriasi et al. 2007. Model evalutaion guidelines for systematic quantification of accuracy in watershed simulations. Transactions of the ASABE 50(3): 885-900.

参数:

• obsvalues – observe values array
• simvalues – simulate values array

Examples

>>> obs = [2.92, 2.75, 2.01, 1.09, 2.87, 1.43, 1.96,                       4.00, 2.24, 29.28, 5.88, 0.86, 13.21]
>>> sim = [2.90, 2.87, 2.85, 2.83, 3.04, 2.81, 2.85,                       2.78, 2.76, 13.40, 2.70, 2.09, 1.62]
>>> MathClass.rsr(obs, sim)  
0.7404026155824978...

返回:RSR value, or raise exception

pygeoc.utils.PI = 3.141592653589793

approximate value of pi.

pygeoc.utils.SQ2 = 1.4142135623730951

approximate of square root of 2.

class pygeoc.utils.StringClass

Bases: object

String handling class

static convert_str2num(unicode_str)

Convert string to string, integer, or float. Support tuple or list.

Examples

>>> StringClass.convert_str2num('1.23')
1.23
>>> StringClass.convert_str2num(u'1.23')
1.23
>>> StringClass.convert_str2num(u'21.')
21
>>> StringClass.convert_str2num('abc123')
'abc123'
>>> StringClass.convert_str2num((123, u'2.3', 3., 'abc', u'edf'))
(123, 2.3, 3, 'abc', 'edf')
>>> StringClass.convert_str2num([123, u'2.3', 3., 'abc', u'edf'])
[123, 2.3, 3, 'abc', 'edf']

static extract_numeric_values_from_string(str_contains_values)

Find numeric values from string, e.g., 1, .7, 1.2, 4e2, 3e-3, -9, etc.

Reference: how-to-extract-a-floating-number-from-a-string-in-python

Examples

>>> input_str = '.1 .12 9.1 98.1 1. 12. 1 12'
>>> StringClass.extract_numeric_values_from_string(input_str)
[0.1, 0.12, 9.1, 98.1, 1, 12, 1, 12]
>>> input_str = '-1 +1 2e9 +2E+09 -2e-9'
>>> StringClass.extract_numeric_values_from_string(input_str)
[-1, 1, 2000000000, 2000000000, -2e-09]
>>> input_str = 'current level: -2.03e+2db'
>>> StringClass.extract_numeric_values_from_string(input_str)
[-203]

参数:str_contains_values – string which may contains numeric values 返回:list of numeric values

static get_datetime(formatted_str, user_fmt=None)

get datetime() object from string formatted %Y-%m-%d %H:%M:%S

Examples

>>> StringClass.get_datetime('2008-11-9')
datetime.datetime(2008, 11, 9, 0, 0)
>>> StringClass.get_datetime('2008/11/9')
datetime.datetime(2008, 11, 9, 0, 0)
>>> StringClass.get_datetime('20081109')
datetime.datetime(2008, 11, 9, 0, 0)
>>> StringClass.get_datetime('11/9/2008')
datetime.datetime(2008, 11, 9, 0, 0)
>>> StringClass.get_datetime('11-9-2008')
datetime.datetime(2008, 11, 9, 0, 0)
>>> StringClass.get_datetime('11/09/08')
datetime.datetime(2008, 11, 9, 0, 0)
>>> StringClass.get_datetime('2008-11-9 11:09')
datetime.datetime(2008, 11, 9, 11, 9)
>>> StringClass.get_datetime('2008-11-9 11:09:52')
datetime.datetime(2008, 11, 9, 11, 9, 52)

static is_substring(substr, str_src)

Is substr part of str_src, case insensitive.

static is_valid_ip_addr(address)

Check the validation of IP address

static split_string(str_src, spliters=None, elim_empty=False)

Split string by split character space(‘ ‘) and indent(‘ ‘) as default

Examples

>>> StringClass.split_string('exec -ini test.ini', ' ')
['exec', '-ini', 'test.ini']

参数:

• str_src – source string
• spliters – e.g. [‘ ‘, ‘ ‘], [], ‘ ‘, None
• elim_empty – Eliminate empty (i.e., ‘’) or not.

返回:

split sub-strings as list

static string_in_list(tmp_str, strlist)

Is tmp_str in strlist, case insensitive.

static string_match(str1, str2)

Compare two string regardless capital or not

class pygeoc.utils.UtilClass

Bases: object

Other common used utility functions

static current_path(local_function)

Get current path, refers to how-do-i-get-the-path-of-the-current-executed-file-in-python

Examples

from pygeoc.utils import UtilClass
curpath = UtilClass.current_path(lambda: 0)

static decode_strs_in_dict(unicode_dict)

Decode strings in dictionary which may contains unicode strings or numeric values.

• 1. integer could be key, float cannot;
• 2. the function is called recursively

Examples

input = {u'name': u'zhulj', u'age': u'28', u'1': ['1', 2, 3]}
output = {'name': 'zhulj', 'age': 28, 1: [1, 2, 3]}
input = {u'name': u'zhulj', 'edu': {'nwsuaf': 2007, u'bnu': '2011', 'igsnrr': 2014}}
output = {'name': 'zhulj', 'edu': {'nwsuaf': 2007, 'bnu': 2011, 'igsnrr': 2014}}

static error(msg)

throw RuntimeError exception

static mkdir(dir_path)

Make directory if not existed

static print_msg(contentlist)

concatenate message list as single string with line feed.

static rmmkdir(dir_path)

If directory existed, then remove and make; else make it.

static run_command(commands)

Execute external command, and return the output lines list. In windows, refers to handling-subprocess-crash-in-windows.

参数:commands – string or list 返回:output lines

static writelog(logfile, contentlist, mode='replace')

write log

pygeoc.utils.ZERO = 1e-12

approximate of zero.

pygeoc.utils.get_config_file()

Get model configuration file name from argv

pygeoc.utils.get_config_parser()

Get config parser.

pygeoc.utils.is_integer(v)

Test whether a value is an integer (of any kind).

Examples

>>> is_integer(1)
True
>>> is_integer(-0.123)
False
>>> is_integer(3.)
False
>>> is_integer(9223372036854775808)
True
>>> is_integer('1')
False
>>> is_integer(None)
False
>>> is_integer(numpy.int(3))
True

pygeoc.utils.is_string(in_str)

Test Unicode (text) string literal.

Examples

>>> is_string('abc')
True
>>> is_string(u'abc')
True
>>> is_string(u'北京')
True
>>> is_string(123)
False
>>> is_string(None)
False
>>> is_string(['a', 'b'])
False

# Python 3 # >>> is_string(b’avoid considering byte-strings as strings.’) # False