How we bring down the costs with Mock Data Library.

mock data library Jan 19, 2021

Being a Startup in AML Space, there is always a lot of data to ingest and process by Machine learning applications. The data is inaccessible due to the nature of its sensitivity, Hence you generate the data with the help of Data Engineers and then consume it within your applications.

Data Flow to Frontend

The generated data corresponds to a certain Scenario or AML Model. The problem starts arising when multiple models are persisted in multiple databases, pointing to multiple versions of the demo environment. This arrangement means that there would be several instances of Backend servers pointing to different databases. These backend servers should be running 24x7 since stakeholders and clients belong to different timezones and geographies.

The Costs starts adding since myriad Services are running on the cloud.

As a Startup, you would want to lower your expenses on these things as Cloud hosting bills stack up a lot if not cared for.

Enter Mock Data Library:-

The idea is simple:- What if One Demo environment corresponds to One version of Mock Data Library which already has all data sufficient for the Application.


Problems we are solving:-

  • No Backend Dependency
  • Keep multiple versions of Demo environment
  • Refresh Data whenever required.

To keep things simple I started with NPM Package Starter and quickly created an NPM Package.

My config start looking something like the below:-

NPM Package Config Sample JSON

Once the skeleton package was ready,

I added Axios Mock Adapter to the application and linked the Mock Adapter instance with the Library.

MockData Instance and Page API's Instance

Whenever the API is called the Mock Instance will be called instead and a static response from Library would be served.

Library Mock Handler

You can add the Mock Data Library to the Package.json locally via the command

yarn add file:mock_data_amls/

This would install the local package in your package.json and then you can normally start your application.


To keep multiple versions of the demo, you can create several branches of the same version above and replace the Data accordingly.

To change the data at any given point in time, I wrote a quick utility in library that will take a config like this

Python Framework Config

After passing this config, It will fetch all these responses and replace the JSON files.

Build the NPM package by issuing command:-

npm run build

I started using https://verdaccio.org/ which is a lightweight open-source private npm proxy registry and start publishing various versions of the mock library so that whenever data is finalized and needs to be published, the library is published on Verdaccio and can be used anytime.

You package.json will look like this:-

"mock_data_amls": "^1.0.0"

Add your registry to NPM

npm adduser --registry http://HOST:PORT

So that next time you issue the command npm install, The NPM or Yarn will pick mock_data_amls from package.json and install it from your hosted registry.

Now we have removed the Backend Dependency since all the data is served from the mock library itself.

This workflow solved the issue of Backend Dependency, Multiple environments, and ever-changing data requirements.

New Workflow with Mock Data Library

The entire process took around a week's effort. Our AWS server cost was reduced considerably since this application can now be hosted anywhere like S3 Bucket or an EC2 instance with Nginx as Reverse proxy to multiple running static applications. Just point your sub-domains to the same server or S3 Bucket and your Demo environment is Ready.


Tags

Shobhit Jain

Frontend Lead at Tookitaki