Hot and Cold: transferring events

When your Event database grows to hundreds of millions of events, there could be some potential performance hits when Casewhere writes data. To optimize performance for those scenarios, Casewhere supports a feature called "Hot and cold". When the feature is enabled, Casewhere will maintain two collections for storing events:

• Hot collection: Casewhere writes directly to this collection. The performance is significantly improved when Casewhere only has to write to a smaller data collection.
• Cold collection: All events will eventually be stored in this collection. There will be a background job running at a scheduled time to periodically transfer events from Hot to Cold.

Enable Hot and Cold

Configure Casewhere deployments

In the Casewhere Admin (project Casewhere.Configuration.Web), update the bellow setting in web.config:

<add key="events:EnableHotAndCold" value="true" />

Similarly, enable the same setting in the web.config of the Worker API (project Casewhere.Worker.Api).

If your deployment is using Azure App Service, just add the key events:EnableHotAndCold in the Application Settings.

Set up the background job for transferring events

Step 1. Download and install

Download the tool here (login is required). After that, copy the tool to the server where you want it to run. The tool is lightweight, so to save cost, you can just run it on the same server with the Web Server or the Database Server.

Install the following dependencies on the server:

1. https://download.visualstudio.microsoft.com/download/pr/69a4b292-6680-46b6-a067-eaf5b4a7ed1e/a14fa32031441812288c113bee953c8d/aspnetcore-runtime-5.0.10-win-x64.exe

2. https://download.visualstudio.microsoft.com/download/pr/9f93e474-1bad-4c66-8445-57a2ef6983d6/b49e4837fc4851a0c85da41d70495c13/dotnet-runtime-5.0.10-win-x64.exe

3. https://download.visualstudio.microsoft.com/download/pr/a0da9621-68f0-439a-b617-4697ee0669e3/38eb4aa6e879b9f06b73599ea2e1535f/dotnet-hosting-5.0.10-win.exe

Step 2. Configure the appsettings.json

1. Update the MongoDBSettings
   1. ConnectionString: The connection string of the **Casewhere Events **database
   2. DatbaseName: The name of Casewhere Events database
   3. SourceEventCollectionName: The Hot collection. The value must be HotEvents
   4. TargetEventCollectionName: The Cold collection. The value must be Events
2. Configure logging at Serilog\WriteTo\Args
   1. Update databaseUrl using the same connection string of the Casewhere Log database used in your deployment.
3. Configure the transferring parameters
   1. BatchSize: The number of events transferred per batch. Default is 1000
   2. MaxDuration: The tool will stop after a configurable period of time. Default is 01:00:00
   3. DelayDurationWhenNoEvents: The tool will keep listen for new events to transfer until it stops. The default delay is 00:00:05

Step 3. Schedule the task

Edit the CreateEventTransferScheduler.ps1 script to:

• Set the $path parameter to the location of the tool
• Provide the $password of the current user, i.e., the user account you use to run this script
• (Optional) Change the $runTime parameter. Default is 3AM

Verify the result in Task Scheduler. Make sure the task will run regardless of whether the user logins or not.

Step 4. Monitor the tool

When the tool runs, it will write log to Casewhere database so you can monitor it using Log Viewer.