Skip to content

Testing Medicare Processing Workflow

General notes Medicare processing workflow

Workflow Documentation

The example below shows how to run the full Medicare processing workflow using the sandbox (test) database. The process takes approximately 15–20 minutes.

Use Singularity shell on a worker node

Testing must be performed in the Dorieh Singularity shell. Please consult Singularity shell section of the "Getting Started" page.

Preparation for Testing

Start the shell

Connect to a worker node in the dorieh partition first!

Ensure you are connected to a reserved worker node before starting the Singularity shell.

Start Dorieh Singularity shell as described in Singularity shell section

Check current database content

Before running the process, check the content of sandbox database:

python -m dorieh.platform.util.psql  --connection  sandbox --db database.ini  "select     public.count_rows('medicare', 'beneficiaries') As NumBeneficiaries,     public.count_rows('medicare', 'enrollments') As CountEnrollments,     public.get_year('medicare'::varchar, 'enrollments'::varchar) As EnrollmentYears,     public.count_rows('medicare', 'admissions') As NumAdmissions,     public.count_rows('medicare_audit', 'admissions') As InvalidAdmissions,     public.get_year('medicare'::varchar, 'admissions'::varchar) As AdmissionsYears,     public.get_year('medicare_audit'::varchar, 'admissions'::varchar) InvalidAdmissionYears;"

 python -m dorieh.platform.util.psql  --connection sandbox --db database.ini 'SELECT year, count(*) from medicare.enrollments GROUP BY year order by year' 

 python -m dorieh.platform.util.psql  --connection sandbox --db database.ini 'SELECT year, count(*) from medicare.admissions GROUP BY year order by year' 

 python -m dorieh.platform.util.psql  --connection sandbox --db database.ini 'SELECT year, count(*) from medicare_audit.admissions GROUP BY year order by year'

Note or save the output into some files.

Test the processing (without ingestion of the new data)

The process can be run in /home/dorieh directory inside singularity shell. If it is run in a different directory, use the absolute path to the database.ini for the --database option.

cd /home/dorieh
nohup toil-cwl-runner --retryCount 1 --cleanWorkDir never --outdir outputs --workDir work --jobStore j1 /usr/local/lib/python3.9/site-packages/dorieh/cwl/medicare.cwl --database database.ini --connection_name sandbox --input empty  2>&1 > 1sandbox.log &
tail -f  1sandbox.log

Examine the file 1sandbox.log to check for errors. Successful completion is indicated by exit code 0 and the absence of “Exception” or “Traceback” in the log.

The process should take about 15 to 20 minutes. Repeat the queries above to test that the content of the sandbox database remains the same. If the workflow starts but does not complete successfully, the database content will change unexpectedly.

Test data ingestion (adding new data)

Data directory structure

Please see Documentation for how data should be placed on the file system for ingestion.

See also General description

Generating sample data from Production data

Dorieh contains a very simple utility to select random records from FWF files provided by ResDac.

In a Singularity shell run:

python -m dorieh.cms.tools.mcr_create_test_data -h

If a directory (e.g., named large_test_data) contains ResDac data - FTS and DAT files, the following command will create a small randomly selected subset of this data for the purpose of testing. By default, the tool selects about 1% of the data which might be more than is needed for testing and would require more time and resources than necessary. Hence, we would modify the selector parameter to select just 0.1% of the data:

python -m dorieh.cms.tools.mcr_create_test_data --in large_test_data --out test_data --selector 0.001

The small subset will be placed in test_data directory. It is important to note that the directory containing the data to be ingested must be named using the 4-digit year corresponding to the data year.

Generate sample data for 2019-2021

In reality, we need to adjust the directory structure of the raw data to follow the pattern described above. The following commands will do the trick:

mkdir test_data
cd test_data
python -m dorieh.cms.tools.mcr_create_test_data --in /data/labs/nsaph_lab/data_management/import/CMS\ Data\ Request\ DUA\ 19945\ -\ RESDAC-9229/13701_copy/13701/2019/mbsf_2019/ --out 2019 --selector 0.001
python -m dorieh.cms.tools.mcr_create_test_data --in /data/labs/nsaph_lab/data_management/import/CMS\ Data\ Request\ DUA\ 19945\ -\ RESDAC-9229/13701_copy/13701/2019/medpar_2019/ --out 2019 --selector 0.005
python -m dorieh.cms.tools.mcr_create_test_data --in /data/labs/nsaph_lab/data_management/import/CMS\ Data\ Request\ DUA\ 19945\ -\ RESDAC-9229/13701_copy/13701/2020/mbsf_2020/ --out 2020 --selector 0.001
python -m dorieh.cms.tools.mcr_create_test_data --in /data/labs/nsaph_lab/data_management/import/CMS\ Data\ Request\ DUA\ 19945\ -\ RESDAC-9229/13701_copy/13701/2020/medpar_2020/ --out 2020 --selector 0.005
python -m dorieh.cms.tools.mcr_create_test_data --in /data/labs/nsaph_lab/data_management/import/CMS\ Data\ Request\ DUA\ 19945\ -\ RESDAC-9229/13701_copy/13701/2021/mbsf_2021/ --out 2021 --selector 0.001
python -m dorieh.cms.tools.mcr_create_test_data --in /data/labs/nsaph_lab/data_management/import/CMS\ Data\ Request\ DUA\ 19945\ -\ RESDAC-9229/13701_copy/13701/2021/medpar_2021/ --out 2021 --selector 0.005

Running Ingestion

Testing data ingestion is similar to the Test Medicare Processing workflow section, but instead of pointing the workflow to an empty directory, we should now point input to the test_data directory created in the previous step.

Info

We have created an empty directory in the Create working directory: section of the Getting Started page.

nohup toil-cwl-runner --retryCount 1 --cleanWorkDir never --outdir outputs     --workDir work --jobStore j1     /usr/local/lib/python3.9/site-packages/dorieh/cwl/medicare.cwl     --database /home/dorieh/database.ini     --connection_name sandbox     --input test_data      2>&1 > 1test.log &

Verifying how database has been updated

Reissue the queries from Check Database Content section and compare the results to the earlier outputs.

You should see additional years or an increase in record counts in tables like enrollments and admissions, depending on your input data.