Stage 1 : Dashboard Creation
The stage 1 of the SafeSquid Dashboard creation talks about organising the safesquid extended log data in an efficient way by using the RDBMS Terminology.
SafeSquid's extended logs contains 37 fields each carrying information about user, website, filters, policies applied etc.
Which provides enough information to user to understand their traffic, policies applied and take appropriate decision depending upon the reports.
Inorder to organize the data in a way that we can easily store, retrieve and process it, we will use the concept of master tables.
Few Steps needs to be taken to convert a the extended log into a an organized structure of files for better efficiency.
Organising the Data:
1. The extended log file will be first processed and a master table will be created from it, termed as data tables.
2. The data tables hold each column(field) of the extended log file.
i.e their will be 37 master files.
Example: username file, requestTypes file, responsetype file etc and so on
3. What will the master files contain?
The master table with contain each fields unique data with a unique id.
Example
Username file:
Username uniqId
ashish@10.1.1.1 1
uma@10.1.1.2 2
This will help us create a new flat file containing the fields value as the uniq id assigned to that particular data in the master file.
The newly generated file is now much more lightweight and easily searchable.