rdkit.ML.Data.DataUtils module — The RDKit 2025.09.6 documentation

# rdkit.ML.Data.DataUtils module

Utilities for data manipulation

FILE FORMATS:

> - .qdat files contain quantized data suitable for

> > feeding to learning algorithms.

> > The .qdat file, written by _DecTreeGui_, is structured as follows:

> > > 1. Any number of lines which are ignored.

> > > 2. A line containing the string ‘Variable Table’

> > >    any number of variable definitions in the format:

> > >    ‘# Variable_name [quant_bounds]’

> > >    > where ‘[quant_bounds]’ is a list of the boundaries used for quantizing

> > >    >   that variable. If the variable is inherently integral (i.e. not quantized), this can be an empty list.

> > > 3. A line beginning with ‘# —-’ which signals the end of the variable list

> > > 4. Any number of lines containing data points, in the format:

> > >    ‘Name_of_point var1 var2 var3 …. varN’

> > >    all variable values should be integers

> > > Throughout, it is assumed that varN is the result

> - .dat files contain the same information as .qdat files, but the variable values can be anything (floats, ints, strings). These files should still contain quant_bounds!

> - .qdat.pkl file contain a pickled (binary) representation of the data read in. They stores, in order:

>   > 1. A python list of the variable names

>   > 2. A python list of lists with the quantization bounds

>   > 3. A python list of the point names

>   > 4. A python list of lists with the data points

rdkit.ML.Data.DataUtils.BuildDataSet(fileName)

  builds a data set from a .dat file

  Arguments

  > - fileName: the name of the .dat file

  Returns

  > an _MLData.MLDataSet_

rdkit.ML.Data.DataUtils.BuildQuantDataSet(fileName)

  builds a data set from a .qdat file

  Arguments

  > - fileName: the name of the .qdat file

  Returns

  > an _MLData.MLQuantDataSet_

rdkit.ML.Data.DataUtils.CalcNPossibleUsingMap(data, order, qBounds, nQBounds=None, silent=True)

  calculates the number of possible values for each variable in a data set

  Arguments

  > - data: a list of examples

  > - order: the ordering map between the variables in _data_ and _qBounds_

  > - qBounds: the quantization bounds for the variables

  Returns

  > a list with the number of possible values each variable takes on in the data set

  Notes

  > - variables present in _qBounds_ will have their _nPossible_ number read from _qbounds

  > - _nPossible_ for other numeric variables will be calculated

rdkit.ML.Data.DataUtils.CountResults(inData, col=-1, bounds=None)

  #DOC

rdkit.ML.Data.DataUtils.DBToData(dbName, tableName, user='sysdba', password='masterkey', dupCol=-1, what='*', where='', join='', pickleCol=-1, pickleClass=None, ensembleIds=None)

  constructs an _MLData.MLDataSet_ from a database

  Arguments

  > - dbName: the name of the database to be opened

  > - tableName: the table name containing the data in the database

  > - user: the user name to be used to connect to the database

  > - password: the password to be used to connect to the database

  > - dupCol: if nonzero specifies which column should be used to recognize duplicates.

  Returns

  > an _MLData.MLDataSet_

  Notes

  > - this uses Dbase.DataUtils functionality

rdkit.ML.Data.DataUtils.FilterData(inData, val, frac, col=-1, indicesToUse=None, indicesOnly=0)

  #DOC

rdkit.ML.Data.DataUtils.InitRandomNumbers(seed)

  Seeds the random number generators

  Arguments

  > - seed: a 2-tuple containing integers to be used as the random number seeds

  Notes

  > this seeds both the RDRandom generator and the one in the standard Python _random_ module

rdkit.ML.Data.DataUtils.RandomizeActivities(dataSet, shuffle=0, runDetails=None)

  randomizes the activity values of a dataset

  Arguments

  > - dataSet: a _ML.Data.MLQuantDataSet_, the activities here will be randomized

  > - shuffle: an optional toggle. If this is set, the activity values will be shuffled (so the number in each class remains constant)

  > - runDetails: an optional CompositeRun object

  Note

  > - _examples_ are randomized in place

rdkit.ML.Data.DataUtils.ReadGeneralExamples(inFile)

  reads the examples from a .dat file

  Arguments

  > - inFile: a file object

  Returns

  > a 2-tuple containing:

  > > 1. the names of the examples

  > > 2. a list of lists containing the examples themselves

  Note

  > - this attempts to convert variable values to ints, then floats. if those both fail, they are left as strings

rdkit.ML.Data.DataUtils.ReadQuantExamples(inFile)

  reads the examples from a .qdat file

  Arguments

  > - inFile: a file object

  Returns

  > a 2-tuple containing:

  > > 1. the names of the examples

  > > 2. a list of lists containing the examples themselves

  Note

  > because this is reading a .qdat file, it assumed that all variable values are integers

rdkit.ML.Data.DataUtils.ReadVars(inFile)

  reads the variables and quantization bounds from a .qdat or .dat file

  Arguments

  > - inFile: a file object

  Returns

  > a 2-tuple containing:

  > > 1. varNames: a list of the variable names

  > > 2. qbounds: the list of quantization bounds for each variable

rdkit.ML.Data.DataUtils.TakeEnsemble(vect, ensembleIds, isDataVect=False)

      >>> v = [10,20,30,40,50]
      >>> TakeEnsemble(v,(1,2,3))
      [20, 30, 40]
      >>> v = ['foo',10,20,30,40,50,1]
      >>> TakeEnsemble(v,(1,2,3),isDataVect=True)
      ['foo', 20, 30, 40, 1]

rdkit.ML.Data.DataUtils.TextFileToData(fName, onlyCols=None)

  #DOC

rdkit.ML.Data.DataUtils.TextToData(reader, ignoreCols=[], onlyCols=None)

  constructs an _MLData.MLDataSet_ from a bunch of text #DOC

  > Arguments

  >   - reader needs to be iterable and return lists of elements (like a csv.reader)

  > Returns

  > > an _MLData.MLDataSet_

rdkit.ML.Data.DataUtils.WriteData(outFile, varNames, qBounds, examples)

  writes out a .qdat file

  Arguments

  > - outFile: a file object

  > - varNames: a list of variable names

  > - qBounds: the list of quantization bounds (should be the same length

  >     as _varNames_)

  > - examples: the data to be written

rdkit.ML.Data.DataUtils.WritePickledData(outName, data)

  writes either a .qdat.pkl or a .dat.pkl file

  Arguments

  > - outName: the name of the file to be used

  > - data: either an _MLData.MLDataSet_ or an _MLData.MLQuantDataSet_

rdkit.ML.Data.DataUtils.permutation(nToDo)