qal.dataset package

Submodules

qal.dataset.custom module

Created on Sep 14, 2012

@author: Nicklas Boerjesson

class qal.dataset.custom.CustomDataset[source]

Bases: builtins.object

This is the base class for all (external) data sets in QAL- Note: The fields are named like this to not appear as parameters in automatic generators like sql_xml. TODO: Field names should be considered captions, not references, and should not be used when looping columns.

apply_new_data(_new_data_table, _key_fields, _insert=None, _update=None, _delete=None, _commit=True)[source]

This function applies a new data table unto the existing, matches are made using the key fields.

Parameters:
  • _new_data_table (2D-list) – A two-dimensional list contains the data. Must match the existing column-wise.
  • _key_fields (list) – An array with the indices of the fields that should be used to match source rows to destination rows.
– note:
If there are different data types in the _new_data_table columns and the existing dataset.data_table, they will be considered different and be updated. It is also possible that the keys will not match. So cast these before applying.
as_sql(_db_type)[source]

Generate SQL Either through a union SQL or into a temp table.

cast_text_to_type(_text, _field_idx)[source]
clear_log()[source]
data_table = None

The actual data, in the form of a two-dimensional array

field_names = None

These are, if applicable, the field names of the resulting dataset

field_types = None

The data types of the fields

load()[source]

Load the data

log_delete(_row_key, _row_data, _comment=None)[source]
log_insert(_row_key, _row_data, _comment=None)[source]
log_load(_filename, _comment=None)[source]
log_save(_filename, _comment=None)[source]
log_update_row(_row_key, _old_row, _new_row, _comment=None)[source]
read_resource_settings(_resource)[source]

Load the data

save()[source]

Save the data to the underlying target.

write_resource_settings(_resource)[source]

Load the data

qal.dataset.factory module

Created on Jan 12, 2014

@author: Nicklas Boerjesson

qal.dataset.factory.dataset_from_resource(_resource)[source]

Create a qal dataset from a resource

qal.dataset.flatfile module

Created on Sep 14, 2012

@author: Nicklas Boerjesson

class qal.dataset.flatfile.FlatfileDataset(_delimiter=None, _filename=None, _has_header=None, _csv_dialect=None, _resource=None, _quoting=None, _quotechar=None, _skipinitialspace=None)[source]

Bases: qal.dataset.custom.CustomDataset

This class loads a flat file into an array, self.data_table.

csv_dialect = None

Specifies the csv dialect

delimiter = None

The delimiter, typically ”;”, ”,” or “|”.

escapechar = None

What character is used to escape special characters, typically “”.

field_names = None

The names of the fields(if applicable, see has_header)

filename = None

The name of the file

has_header = None

True if data begins with a header row containing field names.

lineterminator = None

What character is used to indicate the end of a line. typically “n”.

load()[source]

Load data

quotechar = '"'

What character to use for quoting. Normally “””.

quoting = None

To what degree does the file employ quoting, values: * None = No quoting at all, quotes will be treated as values * “MINIMAL” = Quoting is only used when necessary * “ALL” = Quoting is applied used on all fields. * “NONNUMERIC” = Non-numeric fields are quoted.

read_resource_settings(_resource)[source]
save(_save_as=None)[source]

Save data

skipinitialspace = None

True if initial spaces should be disregarded.

write_resource_settings(_resource)[source]

qal.dataset.matrix module

Created on Jan 8, 2012

@author: Nicklas Boerjesson

class qal.dataset.matrix.MatrixDataset[source]

Bases: qal.dataset.custom.CustomDataset

The matrix dataset holds a two-dimensional array of data

as_sql(_db_type)[source]

Generate SQL

dal = None
load()[source]

Load data. Not implemented as it is not needed in the matrix descendant

save()[source]

Save data. Not implemented as it is not needed in the matrix descendant

qal.dataset.rdbms module

Created on Jan 8, 2012

@author: Nicklas Boerjesson

class qal.dataset.rdbms.RDBMSDataset(_resource=None)[source]

Bases: qal.dataset.custom.CustomDataset

The RDMBS Dataset holds a two-dimensional array of data, typically representing a table in a database. If the data set is not a table but based on a more complex query, data will not be possible to apply to it.

dal = None

An instance of the Database Abstraction Layer(DAL)

load()[source]
query = None

If set, and table_name is not, then the SQL statement contained is executed. It is a text string.

Todo

Make it possible for it to be a backend agnostic qal.sql.sql.VerbSelect object parsed from an XML structure.

read_resource_settings(_resource)[source]
save()[source]

Save data. Commits the transaction

table_name = None

If set, all data in table “table_name” is loaded in its entirety into the data_table.

write_resource_settings(_resource)[source]

TODO: The RDBMS resource type is a special case, as it is both a database connection and/or a dataset definition. Should this be?

qal.dataset.spreadsheet module

Created on Dec 28, 2013

@author: Nicklas Boerjesson

class qal.dataset.spreadsheet.SpreadsheetDataset(_filename=None, _has_header=None, _resource=None, _sheet_name=None, _x_offset=None, _y_offset=None)[source]

Bases: qal.dataset.custom.CustomDataset

The Spreadsheet dataset can read data from a spreadsheet, currently only Excel files, and store that data in its.

field_names = None

The names of the fields(if applicable, see has_header).

filename = None

The name of the file.

has_header = None

True if data begins with a header row containing field names.

load()[source]

Load data.

load_xls(_filename, _sheet_name, _x_offset, _y_offset, _header_offset)[source]
load_xlsx(_filename, _sheet_name, _x_offset, _y_offset, _header_offset)[source]
read_resource_settings(_resource)[source]
save(_save_as=None)[source]

Save data

save_xls(_filename, _data_table, _number_of_rows, _number_of_columns, _x_offset, _y_offset, _header_offset, _field_names, _sheet_name)[source]
save_xlsx(_filename, _data_table, _number_of_rows, _number_of_columns, _x_offset, _y_offset, _header_offset, _field_names, _sheet_name)[source]
sheet_name = None

The name of the worksheet to use, typically “Sheet1”.

write_resource_settings(_resource)[source]
x_offset = None

The leftmost column where data is held

y_offset = None

The topmost row where data is held, including the header row.

qal.dataset.spreadsheet.none_to_zero(_value)[source]

qal.dataset.xpath module

Created on Sep 14, 2012

@author: Nicklas Boerjesson

class qal.dataset.xpath.XpathDataset(_filename=None, _rows_xpath=None, _resource=None)[source]

Bases: qal.dataset.custom.CustomDataset

This class implements data formats that are possible to query via XPath. Currently XML and HTML are implemented(XHTML isn’t tested). Untidy HTML will be implemented using the Beautiful Soup library.

field_xpaths = None

A list of XPaths relative to the rows_xpath leading to each field. Ex: [“title”, “id”]

filename = None

The name of the file to be loaded

load(_add_node_ref=None)[source]

Parse file, apply root XPath to iterate over and then collect field data via field_xpaths.

Parameters:add_node_ref (bool) – If set, add an extra column with references to the underlying XML nodes, used to efficiently apply new data.
read_resource_settings(_resource)[source]

Reads settings from a resource

rows_xpath = None

The XPath to the “row” nodes of the data set. Ex: /books/book

save(_save_as=None)[source]

Save the document

write_resource_settings(_resource)[source]

Write settings to a resource

xpath_data_format = None

The data format. Can be either “XML”, “XHTML” or “HTML”.

qal.dataset.xpath.xpath_data_formats()[source]

Module contents

qal.dataset - Data set classes

The dataset package contains different data sets, all inheriting from the :py:class:qal.dataset.custom.custom class. They provide a generalized way of reading and writing to a multitude of data sources. Currenctly, there are: flat file, matrix, XPath(XML/HTML/XHTML) and spreadsheet(Excel-formats/.odt)

copyright:Copyright 2010-2014 by Nicklas Boerjesson
license:BSD, see LICENSE for details.