Clinical Ink Senior Director Enterprise Architecture Alex Doan recently wrote a case study about how the company chose to use the Rockset analytics platform to improve query speed and other performance metrics.
He began by explaining Clinical Ink, which he described as a suite of software used in over a thousand clinical trials to streamline the data collection and management process, with the goal of improving the efficiency and accuracy of trials. The company’s cloud-based electronic data capture system enables clinical trial data from more than 2 million patients across 110 countries to be collected electronically in real-time from sources including electronic health records and wearable devices.
The COVID-19 pandemic forced many clinical trials to go virtual. Thus, Clinical Ink has been a very valuable solution to researchers because of its ability to support remote monitoring and virtual clinical trials.
During the pandemic, clinical trial sponsors came to Clinical Ink with a requirement for a real-time, 360-degree view of patients and their outcomes across the entire study, which encompassed the whole world. And that 360-degree view had to provide both high-level views of data and robust ability to drill down in order to allow clinical teams to keep the worldwide trials on track.
“When the requirements for the new real-time study participant monitoring came to the engineering team, I knew that the current technical stack could not support millisecond-latency complex analytics on real-time data,” Doan wrote. “Amazon OpenSearch, a fork of Elasticsearch used for our application search, was fast but not purpose-built for complex analytics including joins. Snowflake, the robust cloud data warehouse used by our analyst team … saw significant delays and could not meet the performance requirements of the application.”
The Clinical Ink team needed to come up with a new architecture that would support real-time ingest and complex analytics while being resilient.
The requirements for any software that would replace OpenSearch as the serving layer were:
- Real-time streaming ingest: Data changes from DynamoDB need to be visible and queryable in the downstream database within seconds.
- Millisecond-latency complex analytics (including joins): The database must be able to consolidate global trial data on patients into a 360-degree view.
- Highly resilient: The database is designed to maintain availability and minimize data loss in the face of various types of failures and disruptions.
- Scalable: The database is cloud-native and can scale at the click of a button or an API call with no downtime.
Rockset came to the attention of the Clinical Ink team as a replacement for OpenSearch because it supports complex analytics on low-latency data. Unlike OpenSearch, Rockset employs a Converged Index, a combination of a search index and data stored in columns and rows for optimal query performance. It also supports an SQL-based query language, which allowed the team to meet the sponsors’ requirements for complex analytics.
The team’s test of Rockset showed query response times in the range of double-digit milliseconds. Ultimately, they proved that Rockset can scale as needed and offers excellent query performance.
Clinical Ink is now phasing real-time clinical trial monitoring into production as the new operational data hub for clinical teams. “We have been blown away by the speed of Rockset and its ability to support complex filters, joins, and aggregations,” Doan wrote.
Read the full article and technical data here.