Timeline for Database vs Flat files for rarely accessed data
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 21, 2017 at 12:36 | comment | added | Robert Baron | I have done similar things on the advice of previous leads for similar reasons. In every case we gave up on the similar solution as it didnt scale as well as the database. My take home lessons were to not to do what dbs do well, and don't teinvent the wheel. | |
| Nov 20, 2017 at 15:10 | comment | added | Robert Baron | In memory, things are a great deal less complicated. This is where what you propose work work. When you are dealing with disks, you need to deal with disk cache. Assuming the pameters of the database and os are tuned to block size of the disk, btree will be faster than what you have outlined as there will be fewer reads from disk. I am assuming that you have confise efficiency with speed of access. Efficiency can and is defined in different ways . | |
| Nov 19, 2017 at 21:46 | comment | added | Martin Maat | @Robert I am talking about efficiency regarding resource use, thus addressing the scalability question. This is not about gaining a ms or two.. | |
| Nov 19, 2017 at 13:57 | comment | added | Robert Baron | If the entire db could fit into main memory, then your suggestion may have merit. However, accessibg data in a btree is much faster when dealing with data stored on disk. | |
| Nov 19, 2017 at 10:05 | comment | added | Martin Maat | The question translates to "how do I make this scalable". I just provided a couple of options with different trade offs. If you only need the axel without the spokes or the tire, rolling your own "wheel" can be a viable option. | |
| Nov 19, 2017 at 9:53 | comment | added | amon | So you are suggesting that OP reinvents the wheel and creates their own in-application database engine? The algorithm you've sketched is known as binary search. Likely, the present database engine is already using this (or equivalently a binary tree index). I'm not saying writing such a program is impossible or excessively difficult, but it does seem rather unnecessary. Does really remove so much of the “overhead of RDBMS”? Wouldn't in-process databases like SQLite produce a similar benefit (if any exists) without so much effort? | |
| Nov 19, 2017 at 9:34 | history | answered | Martin Maat | CC BY-SA 3.0 |