Data Set Curation

The framework offers a full suite of features to facilitate the curation of data sets of physical properties, including:

  • a significant amount of data filters, including to filter by state, substance composition and chemical functionalities.

and components to

  • easily download and import the full NIST ThermoML and FreeSolv archives .

  • select data points which were measured close to a set of target states, and which were measured for a diverse range of substances which contain specific functionalities.

  • convert between different compatible property types (e.g. convert density <-> excess molar volume data).

These features are implemented as CurationComponent objects, which take as input an associated CurationComponentSchema which controls how the curation components should be applied to a particular data set (or a data set which is being stored as pandas DataFrame object).

An example of a curation component would be one that filters out data points which were measured outside of a particular temperature range:

# Filter data points measured at less than 290.0 K or greater than 320.0 K
filtered_frame = FilterByTemperature.apply(
    data_frame,
    FilterByTemperatureSchema(minimum_temperature=290.0, maximum_temperature=320.0),
)

Curation components can be conveniently chained together using a CurationWorkflow and an associated CurationWorkflowSchema so as to easily curated full training and testing data sets:

curation_schema = WorkflowSchema(
    component_schemas=[
        # Import the ThermoML archive.
        thermoml.ImportThermoMLDataSchema()
        # Filter out any measurements made for systems with more than two components
        filtering.FilterByNComponentsSchema(n_components=[1, 2]),
        # Remove any duplicate data.
        filtering.FilterDuplicatesSchema(),
        # Filter out data points measured away from ambient
        # and biologically relevant temperatures.
        filtering.FilterByTemperatureSchema(
            minimum_temperature=298.0, maximum_temperature=320.0
        ),
        # Retain only density and enthalpy of mixing data points.
        filtering.FilterByPropertyTypesSchema(
            property_types=["Density", "EnthalpyOfMixing"],
        ),
        # Select data points measured for alcohols, esters or mixtures of both.
        selection.SelectSubstancesSchema(
            target_environments=[
                ChemicalEnvironment.Alcohol,
                ChemicalEnvironment.CarboxylicAcidEster,
            ],
            n_per_environment=10,
        ),
    ]
)

data_frame = Workflow.apply(pandas.DataFrame(), curation)

Examples

Data Extraction

  • ImportFreeSolv: A component which will download the latest, full FreeSolv data set from the GitHub repository:

    from openff.evaluator.datasets.curation.components.freesolv import (
        ImportFreeSolv,
        ImportFreeSolvSchema,
    )
    
    # Import the full FreeSolv data set.
    data_frame = ImportFreeSolv.apply(pandas.DataFrame(), ImportFreeSolvSchema())
    
  • ImportThermoMLData: A component which will download all supported data from the NIST ThermoML Archive:

    from openff.evaluator.datasets.curation.components.thermoml import (
        ImportThermoMLData,
        ImportThermoMLDataSchema,
    )
    
    # Import all data collected from the IJT journal.
    data_frame = ImportThermoMLData.apply(
        pandas.DataFrame(), ImportThermoMLDataSchema(journal_names=["IJT"])
    )
    

Filtration

  • FilterDuplicates: A component to remove duplicate data points (within a specified precision) from a data set:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterDuplicates,
        FilterDuplicatesSchema,
    )
    
    filtered_frame = FilterDuplicates.apply(data_frame, FilterDuplicatesSchema())
    
  • FilterByTemperature: A component which will filter out data points which were measured outside of a specified temperature range:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByTemperature,
        FilterByTemperatureSchema,
    )
    
    filtered_frame = FilterByTemperature.apply(
        data_frame,
        FilterByTemperatureSchema(minimum_temperature=290.0, maximum_temperature=320.0),
    )
    
  • FilterByPressure: A component which will filter out data points which were measured outside of a specified pressure range:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByPressure,
        FilterByPressureSchema,
    )
    
    filtered_frame = FilterByPressure.apply(
        data_frame,
        FilterByPressureSchema(minimum_pressure=100.0, maximum_pressure=140.0),
    )
    
  • FilterByMoleFraction: A component which will filter out data points which were measured outside of a specified mole fraction range:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByMoleFraction,
        FilterByMoleFractionSchema,
    )
    
    filtered_frame = FilterByMoleFraction.apply(
        data_frame, FilterByMoleFractionSchema(mole_fraction_ranges={2: [[(0.1, 0.3)]]})
    )
    
  • FilterByRacemic: A component which will filter out data points which were measured for racemic mixtures:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByRacemic,
        FilterByRacemicSchema,
    )
    
    filtered_frame = FilterByRacemic.apply(data_frame, FilterByRacemicSchema())
    
  • FilterByElements: A component which will filter out data points which were measured for systems which contain specific elements:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByElements,
        FilterByElementsSchema,
    )
    
    filtered_frame = FilterByElements.apply(
        data_frame,
        FilterByElementsSchema(allowed_elements=["C", "O", "H"]),
    )
    
  • FilterByPropertyTypes: A component which will apply a filter which only retains properties of specified types:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByPropertyTypes,
        FilterByPropertyTypesSchema,
    )
    
    # Retain only density measurements made for either pure or binary systems.
    filtered_frame = FilterByPropertyTypes.apply(
        data_frame,
        FilterByPropertyTypesSchema(
            property_types=["Density"],
            n_components={"Density": [1, 2]},
        ),
    )
    
  • FilterByStereochemistry: A component which filters out data points measured for systems whereby the stereochemistry of a number of components is undefined:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByStereochemistry,
        FilterByStereochemistrySchema,
    )
    
    filtered_frame = FilterByStereochemistry.apply(
        data_frame, FilterByStereochemistrySchema()
    )
    
  • FilterByCharged: A component which filters out data points measured for substance where any of the constituent components have a net non-zero charge.:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByCharged,
        FilterByChargedSchema,
    )
    
    filtered_frame = FilterByCharged.apply(data_frame, FilterByChargedSchema())
    
  • FilterByIonicLiquid: A component which filters out data points measured for substances which contain or are classed as an ionic liquids:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByIonicLiquid,
        FilterByIonicLiquidSchema,
    )
    
    filtered_frame = FilterByIonicLiquid.apply(data_frame, FilterByIonicLiquidSchema())
    
  • FilterBySmiles: A component which filters the data set so that it only contains either a specific set of smiles, or does not contain any of a set of specifically excluded smiles:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterBySmiles,
        FilterBySmilesSchema,
    )
    
    filtered_frame = FilterBySmiles.apply(
        data_frame, FilterBySmilesSchema(smiles_to_include=["CCCO"]),
    )
    
  • FilterBySmirks: A component which filters a data set so that it only contains measurements made for molecules which contain (or don’t) a set of chemical environments represented by SMIRKS patterns:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterBySmirks,
        FilterBySmirksSchema,
    )
    
    filtered_frame = FilterBySmirks.apply(
        data_frame, FilterBySmirksSchema(smirks_to_include=["[#6a]"]),
    )
    
  • FilterByNComponents: A component which filters out data points measured for systems with specified number of components:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByNComponents,
        FilterByNComponentsSchema,
    )
    
    filtered_frame = FilterByNComponents.apply(
        data_frame, FilterByNComponentsSchema(n_components=[1, 2])
    )
    
  • FilterBySubstances: A component which filters the data set so that it only contains properties measured for particular substances:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterBySubstances,
        FilterBySubstancesSchema,
    )
    
    filtered_frame = FilterBySubstances.apply(
        data_frame, FilterBySubstancesSchema(substances_to_include=[("CO", "C")])
    )
    
  • FilterByEnvironments: A component which filters a data set so that it only contains measurements made for substances which contain specific chemical environments:

    from openff.evaluator.datasets.curation.components.filtering import (
        FilterByEnvironments,
        FilterByEnvironmentsSchema,
    )
    
    filtered_frame = FilterByEnvironments.apply(
        data_frame,
        FilterByEnvironmentsSchema(
            environments=[
                ChemicalEnvironment.Aqueous,
                ChemicalEnvironment.Alcohol,
                ChemicalEnvironment.Amine,
            ]
        ),
    )
    

Data Selection

  • SelectSubstances: A component for selecting data points which were measured for specified number of maximally diverse systems containing a specified set of chemical functionalities:

    # Select (if possible) data points which were measured for 10 different (and
    # structurally diverse) alcohols.
    schema = SelectSubstancesSchema(
        target_environments=[ChemicalEnvironment.Alcohol],
        n_per_environment=10,
    )
    
    data_frame = ConvertExcessDensityData.apply(data_frame, schema)
    
  • SelectDataPoints: A component for selecting a set of data points which are close to a particular set of states:

    # Select (if possible) density data points which were measured for pure systems
    # at close to 298.15 K and 308.15K
    schema = SelectDataPointsSchema(
        target_states=[
            TargetState(
                property_types=[("Density", 1)],
                states=[
                    State(temperature=298.15, pressure=101.325, mole_fractions=(1.0,),
                    State(temperature=308.15, pressure=101.325, mole_fractions=(1.0,),
                ],
            )
        ]
    )
    
    data_frame = ConvertExcessDensityData.apply(data_frame, schema)
    

Data Conversion

  • ConvertExcessDensityData: A component for converting binary mass density data to excess molar volume data and vice versa where pure density data measured for the components is available:

    from openff.evaluator.datasets.curation.components.conversion import (
        ConvertExcessDensityData,
        ConvertExcessDensityDataSchema,
    )
    
    converted_data_frame = ConvertExcessDensityData.apply(
        data_frame, ConvertExcessDensityDataSchema()
    )