EMMAA Model Test Framework (emmaa.model_tests)

This module implements the object model for EMMAA model testing.

class emmaa.model_tests.EmmaaTest[source]

Bases: object

Represent an EMMAA test condition

get_entities()[source]

Return a list of entities that the test checks for.

class emmaa.model_tests.ModelManager(model, mode='local')[source]

Bases: object

Manager to generate and store properties of a model and relevant tests.

Parameters

  • model (emmaa.model.EmmaaModel) – EMMAA model

  • mode (str) – If ‘local’ (default), does not save any exports/images to S3. It is only set to ‘s3’ mode in update_model_manager.py script.

mc_mapping

A dictionary mapping a ModelChecker type to a corresponding method for assembling the model and a ModelChecker class.

Type

dict

mc_types

A dictionary in which each key is a type of a ModelChecker and value is a dictionary containing an instance of a model, an instance of a ModelChecker and a list of test results.

Type

dict

entities

A list of entities of EMMAA model.

Type

list[indra.statements.agent.Agent]

applicable_tests

A list of EMMAA tests applicable for given EMMAA model.

Type

list[emmaa.model_tests.EmmaaTest]

date_str

Time when this object was created.

Type

str

path_stmt_types

A dictionary mapping statement hashes to a count of paths they are in.

Type

dict

add_result(mc_type, result)[source]

Add a result to a list of results.

add_test(test)[source]

Add a test to a list of applicable tests.

answer_dynamic_query(query, bucket='emmaa')[source]

Answer user query by simulating a PySB model.

answer_intervention_query(query, bucket='emmaa')[source]

Answer user intervention query by simulating a PySB model.

answer_open_query(query)[source]

Answer user open search query with found paths.

answer_path_query(query)[source]

Answer user query with a path if it is found.

answer_queries(queries, **kwargs)[source]

Answer all queries registered for this model.

Parameters

queries (list[emmaa.queries.Query]) – A list of queries to run.

Returns

responses – A list of tuples each containing a query, mc_type and result json.

Return type

list[tuple(json, json)]

get_updated_mc(mc_type, stmts, add_ns=False, edge_filter_func=None)[source]

Update the ModelChecker and graph with stmts for tests/queries.

hash_response_list(response)[source]

Return a dictionary mapping a hash with a response in a response list.

process_response(mc_type, result)[source]

Return a dictionary in which every key is a hash and value is a list of tuples. Each tuple contains a sentence describing either a step in a path (if it was found) or result code (if a path was not found) and a link leading to a webpage with more information about corresponding sentence.

results_to_json(test_data=None)[source]

Put test results to json format.

run_all_tests(filter_func=None, edge_filter_func=None, allow_direct=True)[source]

Run all applicable tests with all available ModelCheckers.

run_tests_per_mc(mc_type, max_path_length, max_paths, filter_func=None, edge_filter_func=None, allow_direct=True)[source]

Run all applicable tests with one ModelChecker.

save_assembled_statements(upload_to_db=True, bucket='emmaa')[source]

Upload assembled statements jsons to S3 bucket.

upload_results(test_corpus='large_corpus_tests', test_data=None, bucket='emmaa')[source]

Upload results to s3 bucket.

class emmaa.model_tests.RefinementTestConnector[source]

Bases: TestConnector

Determines applicability of a test to a model by checking if test entities or their refinements are in the model.

static applicable(model, test)[source]

Return True of all test entities are in the set of model entities

class emmaa.model_tests.ScopeTestConnector[source]

Bases: TestConnector

Determines applicability of a test to a model by overlap in scope.

static applicable(model, test)[source]

Return True of all test entities are in the set of model entities

class emmaa.model_tests.StatementCheckingTest(stmt, configs=None)[source]

Bases: EmmaaTest

Represent an EMMAA test condition that checks a PySB-assembled model against an INDRA Statement.

check(model_checker, pysb_model)[source]

Use a model checker to check if a given model satisfies the test.

get_entities()[source]

Return a list of entities that the test checks for.

class emmaa.model_tests.TestConnector[source]

Bases: object

Determines if a given test is applicable to a given model.

static applicable(model, test)[source]

Return True if the test is applicable to the given model.

class emmaa.model_tests.TestManager(model_managers, tests)[source]

Bases: object

Manager to generate and run a set of tests on a set of models.

Parameters
make_tests(test_connector)[source]

Generate a list of applicable tests for each model with a given test connector.

Parameters

test_connector (emmaa.model_tests.TestConnector) – A TestConnector object to use for connecting models to tests.

run_tests(filter_func=None, edge_filter_func=None, allow_direct=True)[source]

Run tests for a list of model-test pairs

emmaa.model_tests.load_tests_from_s3(test_name, bucket='emmaa')[source]

Load Emmaa Tests with the given name from S3.

Parameters

test_name (str) – Looks for a test file in the emmaa bucket on S3 with key ‘tests/{test_name}’.

Returns

List of EmmaaTest objects loaded from S3.

Return type

list of EmmaaTest

emmaa.model_tests.model_to_tests(model_name, upload=True, bucket='emmaa')[source]

Create StatementCheckingTests from model statements.

emmaa.model_tests.run_model_tests_from_s3(model_name, test_corpus='large_corpus_tests', upload_results=True, bucket='emmaa')[source]

Run a given set of tests on a given model, both loaded from S3.

After loading both the model and the set of tests, model/test overlap is determined using a ScopeTestConnector and tests are run.

Parameters

  • model_name (str) – Name of EmmaaModel to load from S3.

  • test_corpus (str) – Name of the file containing tests on S3.

  • upload_results (Optional[bool]) – Whether to upload test results to S3 in JSON format. Can be set to False when running tests. Default: True

Returns

Instance of ModelManager containing the model data, list of applied tests and the test results.

Return type

emmaa.model_tests.ModelManager

emmaa.model_tests.save_tests_to_s3(tests, bucket, key, save_format='pkl')[source]

Save tests in pkl, json or jsonl format.