2012-2013
Web: http://apps.njmeadowlands.gov/chemistry/sediment/
Abstract
The New Jersey Meadowlands Commission (NJMC) has been funding and preforming environmental studies within the New Jersey Meadowlands Region as far back as 1970. The NJMC houses the findings from all these studies within the Meadowlands Environmental Research Institute’s (MERI) library. Scientists interested in historical environmental data for the region have to search through hundreds of studies and thousands of sample records to find what they are looking for.
MERI’s Sediment Chemistry Data Portal serves as a way to easily access MERI’s rich archive of historical sediment chemistry data. The portal allows users to search by chemical, study, or Chemical in a particular study. Query results are displayed in a tabular form featuring chemical information, geographic coordinates, and sampling dates. Results can easily be exported to excel for further analysis.
System Overview
The Sediment Chemistry Data Portal (SCDP) features a large multi-relationship database that stores information about chemicals, sampling information, geographic coordinates, and sample depth information. Though over the years many different methods sampling methods were used to record data, this system unifies all data through similar criteria and related data tables. Algorithms within the system are used to query data and return accurate result sets. Since the database is quite large (over 190,000 samples) each query is cached by the system to reduce loading times and improve overall performance. When new studies are ingested into the system via the Sediment Chemistry Data Importing Application all caches are deleted, and popular caches are dynamically rebuilt.
Data Entry
Summer interns prepared historical studies in Microsoft Excel spread sheets. Many of these studies already existed in a spread sheet format, and much of the interns’ work focused on quality control and review. Older studies that were in paper form only were scanned by interns and entered into the Sediment Chemistry Data Importing Application manually.
Sediment Chemistry Data Importing Application
In order to facilitate a large quantity of data entry an application was written to read and process each study in spreadsheet form. To do this the application allows users to open a spreadsheet in the program and then answer a few questions about what each column and row belongs to. This application greatly increased data ingestion speeds and allowed MERI to process about 50,000 records into the database per week. To protect against faulty data or user mistakes each process receives a unique identifier that can be used to remove faulty records from the database without harming the integrity of the remaining data.
Credits
New Jersey Meadowlands Commission Staff
Application & Database Development: Brian Kennedy
Data Entry Staff: Trevor DiLallo, Sachin Jain