AWS model update and testing pipeline (emmaa.aws_lambda_functions)

The AWS Lambda emmaa-update-pipeline definition.

This file contains the function that starts model update cycle. It must be placed on AWS Lambda, which can either be done manually (not recommended) or by running:

$ python update_lambda.py update_pipeline.py emmaa-update-pipeline

in this directory.

emmaa.aws_lambda_functions.update_pipeline.lambda_handler(event, context)

Invoke individual model update functions.

This function iterates through all models contained on S3 bucket and calls a different Lambda function to run model update for the models configured to be updated daily. It is expected that models have ‘run_model_update’ parameter in their config.json files.

This function is designed to be placed on AWS Lambda, taking the event and context arguments that are passed. Note that this function must always have the same parameters, even if any or all of them are unused, because we do not have control over what Lambda sends as parameters. Parameters are unused in this function.

Lambda is configured to automatically run this script every day.

See the top of the page for the Lambda update procedure.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A response returned by the latest call to emmaa-model-update function.

Return type:

dict

The AWS Lambda emmaa-model-update definition.

This file contains the function that starts model update cycle. It must be placed on AWS Lambda, which can either be done manually (not recommended) or by running:

$ python update_lambda.py model_updates.py emmaa-model-update

in this directory.

emmaa.aws_lambda_functions.model_updates.lambda_handler(event, context)

Create a batch job to update models on s3 and NDEx.

This function is designed to be placed on AWS Lambda, taking the event and context arguments that are passed. Note that this function must always have the same parameters, even if any or all of them are unused, because we do not have control over what Lambda sends as parameters. Event parameter is used to pass model_name argument.

This Lambda function is configured to be invoked by emmaa-update-pipeline Lambda function.

See the top of the page for the Lambda update procedure.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case the dictionary contains model name.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, ‘result’, and ‘job_id’ to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-mm-update definition.

This file contains the function that updates model manager object in S3. It must be placed on AWS Lambda, which can either be done manually (not recommended) or by running:

$ python update_lambda.py model_manager_update.py emmaa-mm-update

in this directory.

emmaa.aws_lambda_functions.model_manager_update.lambda_handler(event, context)

Create a batch job to update model manager on s3.

This function is designed to be placed on AWS Lambda, taking the event and context arguments that are passed. Note that this function must always have the same parameters, even if any or all of them are unused, because we do not have control over what Lambda sends as parameters. This Lambda function is configured to be triggered when the model is updated on S3.

See the top of the page for the Lambda update procedure.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case, we are expecting ‘Records’, each of which contains a record of a file that was added (or changed) on s3.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, ‘result’, and ‘job_id’ to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-after-update definition.

This file contains the function that will be run when Lambda is triggered. It must be placed on s3, which can either be done manually (not recommended) or by running:

$ python update_lambda.py after_update.py emmaa-after-update

in this directory.

emmaa.aws_lambda_functions.after_update.lambda_handler(event, context)

Submit model tests, model and test stats, and query batch jobs.

This function is designed to be placed on AWS Lambda, taking the event and context arguments that are passed. Note that this function must always have the same parameters, even if any or all of them are unused, because we do not have control over what Lambda sends as parameters. Event parameter is used here to pass which model manager was updated.

Lambda is configured to run this script when ModelManager object is updated.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case, we are expecting ‘Records’, each of which contains a record of a file that was added (or changed) on s3.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, and any other data to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-test-pipeline definition.

This file contains the function that will be run when Lambda is triggered. It must be placed on s3, which can either be done manually (not recommended) or by running:

$ python update_lambda.py test_pipeline.py emmaa-test-pipeline

in this directory.

emmaa.aws_lambda_functions.test_pipeline.lambda_handler(event, context)

Invoke individual test corpus functions.

This function is designed to be placed on lambda, taking the event and context arguments that are passed. Event parameter is used here to pass name of the model.

This Lambda function is configured to be invoked by emmaa-after-update Lambda function.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case the dictionary contains ‘model’ key.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A response returned by the latest call to emmaa-model-test function.

Return type:

dict

The AWS Lambda emmaa-model-test definition.

This file contains the function that will be run when Lambda is triggered. It must be placed on s3, which can either be done manually (not recommended) or by running:

$ python update_lambda.py model_tests.py emmaa-model-test

in this directory.

emmaa.aws_lambda_functions.model_tests.lambda_handler(event, context)

Create a batch job to run model tests.

This function is designed to be placed on lambda, taking the event and context arguments that are passed. Event parameter is used here to pass names of the model and of the test corpus.

This Lambda function is configured to be invoked by emmaa-test-pipeline Lambda function.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case the dictionary contains ‘model’ and ‘tests’ keys.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, and any other data to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-test-stats definition.

This file contains the function that will be run when Lambda is triggered. It must be placed on s3, which can either be done manually (not recommended) or by running:

$ python update_lambda.py test_stats.py emmaa-test-stats

in this directory.

emmaa.aws_lambda_functions.test_stats.lambda_handler(event, context)

Create a batch job to generate model statistics.

This function is designed to be placed on lambda, taking the event and context arguments that are passed, and extracting the names of the uploaded (which includes changed) model or test definitions on s3. Lambda is configured to be triggered by any such changes, and will automatically run this script.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case, we are expecting ‘Records’, each of which contains a record of a file that was added (or changed) on s3.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, and any other data to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-model-stats definition.

This file contains the function that will be run when Lambda is triggered. It must be placed on s3, which can either be done manually (not recommended) or by running:

$ python update_lambda.py model_stats.py emmaa-model-stats

in this directory.

emmaa.aws_lambda_functions.model_stats.lambda_handler(event, context)

Create a batch job to generate model statistics.

This function is designed to be placed on lambda, taking the event and context arguments that are passed. Event parameter is used here to pass name of the model.

This Lambda function is configured to be invoked by emmaa-after-update Lambda function.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case the dictionary contains ‘model’ key.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, and any other data to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-queries definition.

This file contains the function that will be run when Lambda is triggered. It must be placed on s3, which can either be done manually (not recommended) or by running:

$ python update_lambda.py model_queries.py emmaa-queries

in this directory.

emmaa.aws_lambda_functions.model_queries.lambda_handler(event, context)

Create a batch job to run queries for model.

This function is designed to be placed on lambda, taking the event and context arguments that are passed. Event parameter is used here to pass name of the model.

This Lambda function is configured to be invoked by emmaa-after-update Lambda function.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case the dictionary contains ‘model’ key.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, and any other data to be returned to Lambda.

Return type:

dict

The AWS Lambda emmaa-test-update-pipeline definition.

This file contains the function that starts model update cycle. It must be placed on AWS Lambda, which can either be done manually (not recommended) or by running:

$ python update_lambda.py test_update_pipeline.py emmaa-test-update-pipeline

in this directory.

emmaa.aws_lambda_functions.test_update_pipeline.lambda_handler(event, context)

Invoke individual model update functions.

This function iterates through all models contained on S3 bucket and calls a different Lambda function to turn the model into tests if the model is configured to do so. It is expected that models have ‘make_tests’ parameter in their config.json files.

This function is designed to be placed on AWS Lambda, taking the event and context arguments that are passed. Note that this function must always have the same parameters, even if any or all of them are unused, because we do not have control over what Lambda sends as parameters. Parameters are unused in this function.

Lambda is configured to automatically run this script every day.

See the top of the page for the Lambda update procedure.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A response returned by the latest call to emmaa-test-update function.

Return type:

dict

The AWS Lambda emmaa-test-update definition.

This file contains the function that updates tests created from model. It must be placed on AWS Lambda, which can either be done manually (not recommended) or by running:

$ python update_lambda.py test_update.py emmaa-test-update

in this directory.

emmaa.aws_lambda_functions.test_update.lambda_handler(event, context)

Create a batch job to update tests on s3.

This function is designed to be placed on AWS Lambda, taking the event and context arguments that are passed. Note that this function must always have the same parameters, even if any or all of them are unused, because we do not have control over what Lambda sends as parameters. Event parameter is used to pass model_name argument.

See the top of the page for the Lambda update procedure.

Parameters:

• event (dict) – A dictionary containing metadata regarding the triggering event. In this case the dictionary contains model name.
• context (object) – This is an object containing potentially useful context provided by Lambda. See the documentation cited above for details.

Returns:

ret – A dict containing ‘statusCode’, with a valid HTTP status code, ‘result’, and ‘job_id’ to be returned to Lambda.

Return type:

dict

emmaa.aws_lambda_functions.update_lambda.upload_function(script_name, function_name)

Upload the lambda function by pushing a zip file to Lambda.

This function pre-supposes you are running from the same directory that contains the lambda script.

Parameters:

• script_name (str) – Name of a script containing lambda function.
• function_name (object) – Name of a lambda function as specified on AWS Lambda.