Saturday, August 16, 2014

Native File System Based Data Management Architecture

All database systems either relational or non-relational, no-sql or schema-less systems store data in some files on the system. The difference is in the way they manage data in files and facilities they provide on top of their architecture.
Here we are not going to concentrate on what they do and how they manage data, this theory strongly focuses on storing and managing data without using any database server.

You can use file system to store and fetch data as this will be much faster compared to R-DBMS systems and much light on resources compared to heavy non R-DBMS solutions available in the market.
Again this speed and being light on resources depends on the way data is stored and managed in files and across directories. So, it is very important to efficiently use file system and manage data properly in such a way that it makes as much minimal resource usage as possible.
Below presented is the concept for how to efficiently store and manage data and what things need to be taken care of while designing architecture of such data storage system

1) Keep file size minimal
As all data management related operations will be as a part of application mostly it will be in programming language of the developer's choice and so they can feel comfortable while applying various data manipulations.
But, care must be taken that large amount of data is not handled at any single moment of time, like reading a large file synchronously.
To avoid large files data must be stored in multiple files and as much discrete as possible. For e.g. One folder for one table and one file inside that folder for each record.
Key part of a database systems is resource (cpu & memory) management plus operations on data. If your design leads to huge memory usage you are doing misuse of file system.

2) Minimize need of locks and waits to access files
Make a read or write operation as much independent of other read & write operations as possible
If your data stored is across multiple files you can easily access large data and operate on them quickly using concurrent threads. But if it is in single large file you might face data corruption of delay due to locks and waits in case of concurrent access.
As explained in previous point, if each write is done in separate file you can avoid locks and waits for file access which will further speed up reading and writing.
Always give unique name to files

3) Minimize read and write to files (File naming convention & directory structure matters)
Yes, although we are going to store data in files that doesn't mean that all data must be stored as part of file content
Design directory structure and file name in such a way it conveys a relation between data and existence of a file or directory can also denote existence of a relation between data. This will reduce file read and write operations and will make managing relations very easy.
For e.g.:
Inside folder named 'friendship' a file named 'userid1-userid2' can denote friendship between two users.
Now all files starting with 'userid1-' or ending with '-userid1' in that folder can give you all friends of that user with userid1 but that is a bad practice to fetch data in that manner.
Even though we can get friends from 'friendship' folder by matching userid1 from name of files, there can be many file in that folder and going through all files will again slow down process.
Instead, it is better to have one folder named 'friends' inside a folder named 'userid1' and store one file each for each friends of userid1 inside that 'friends' folder.
Thus, when we need to get all list for friends of userid1 we will look into '/data/users/userid1/friends/' folder. While the '/data/friendship/' folder can be used to store and read data related to relation between those two users like status, added date, etc.

4) Managing operations
While applying operations like average of some value, you can read values stored in files using concurrent threads and in more complex case can even use temporary files to store data or manage operations

5) Use Common-Sense
Think before you confirm your flow and data management scheme. Consider amount of simultaneous users and amount of data to be stored. Maximize use of threads.
Although, all data can be stored and managed in file system this approach also has its own limitations like handling complex operations on large data needs an extreme expertise.
Again its my duty to make you aware you that if you need to perform complex logic on huge data this approach might not fit into your need and better consider other options as mentioned above in necessity block of full article on the website PLS-E.IN.

Also, for more details and proof of concept visit PLS-E.IN

Too much theoretical to digest, right ? Checkout our products page for "Simple Web Chat" application which is developed in php and works on same concept !

Thursday, August 2, 2012

Curtailed Concise Classes And Trimmed (Reduced) Objects Of A Class

Every one knows memory management is one of the most crucial aspect in development of software / website applications. Using OOP memory management can be easy done by deallocating memory of an object when that object is no more required. But, a class contains many functions that may not be used all together each and every time object of that class is used.

Consider the case when one uses a function of an object of a class (for a serving a webpage) say one function 'a' which again is dependant on function 'b' & 'c' of that class, so totally 3 functions are of use when that web page is loaded. But when we create object of that class, it will contain all functions defined in that class. So, instead of consuming memory of only 3 functions it will consume memory that is required for all function present in that class (Yes, i know that this memory is not much but there is an old saying that lake is built-up of small water droplets). And everyone knows that functionality, complexity, bulkiness of software / web and mobile applications are increasing at very rapid pace, may be: year by year if not day by day.

So lets get back to our original discussion, to write a class separately for each call (considering web development and only one db model class into account, lets say it be a call (web request) for insert) which requires only functions used in that call (say one is for select to check if record exists and other to insert) it will be very tedious, time taking and doesn't suite true object oriented approach to separate those functions required in to separate classes based upon req. of request, more over this req. for many requests can change over time and again lot of functions from here to there. Also, this doesn't suite a true technical person when it is possible to generate such class / objects dynamically at run-time.

This might seem to increase execution time but only during development and during actual time of generating such classes as generation might take some time (mostly not more than few milliseconds for one class). Once the application is deployed & in live environment there is no need to to re-create those classes unless there is any change in code, as those can be stored and used again & updating them is req. only when original one is updated.

For more details and proof of concept visit PLS-E.IN

Will soon make it available for public use once setup on public internet server and post its link here. So, stay in touch. Hopefully, meet you all soon.

Its now already available for free use (currently supports only php classes) at PLS-E.IN

Saturday, November 7, 2009


APEI : It is a combination of 'File Explorer' & 'Web Browser' with look & feel of Windows, yet completely different and with more user friendly GUI. 

It has many features which otherwise, on a normal windows xp platform would have required installation of many 3rd party softwares.

It works fine on most of xp & vista versions. It is a freeware, so enjoy your desktop experience with it !

(Homepage) [Downloads: Available at PLS-E.IN
(Requirements: 'DotNet Framework 2.0', 'Windows Installer 3.1')

Win Roll

Win Roll : It is a nice, light on resource software which makes opened window to contract only to show its title bar (when right click on the title bar). 

Extremely useful to see back ground window when too many windows are opened on the desktop. 

Its a free & open source software, so make full use of it to boost your productivity. Enjoy !

(Homepage) (Download) (Source Code)

Monday, October 5, 2009


This blog will mostly contains link & posts about various useful and unique stuffs related to computer soft-wares and / or applications.

You can also visit our website PLS-E.IN for more details.