FLASHC

By "we can say", you must mean you and your tapeworm. No one else is included in your conclusion.

 

By "document" you really mean "input data stream". In strict terms, an ElasticSearch "document" is a MySQL "row". You will never succeed in loading a 1 GB row into MySQL. But from your posts, I am guessing that MySQL has a tool that slurps one huge 1 GB input stream into the multiple rows it represents and loads them optimally. OK, ElasticSearch doesn't come with such a tool, but it comes with wonderful APIs that let you dream up and implement all manner of input streams. There are many third-party tools for pulling in data from many sources (rivers, they call them), and I wrote my own converters with proper bulk-load coding to push bulk data into ElasticSearch.

 

I can easily and successfully load a 3.1 GB "document" into ElasticSearch. Even on my laptop with decent CPU power but low end disk performance, I can load this 3.1 GB monster in just under 3 hours. The MacBook fans sound like a (quiet) jet engine, but the system is still surprisingly responsive during its efforts. And there are no memory issues, exceptions thrown, or any other issues at all. And note that this exact same 3.1 GB input "document" was loaded into MySQL in 8 hours on a production server with a proper disk array; ElasticSearch did the same job on my laptop and single slow disk in less than half the time.

 

And that 3.1 GB document is a gzip'd CSV file. Of course, I needed my Java skills to take the gunzip'd output (using gunzip -c to decompress to stdout but not on disk. Yay!), then convert that (probably about 7 or 8 GB by now) uncompressed CSV stream into the desired JSON stream, and the use the excellent examples as a model for my bulk loader that properly loaded that huge document into ElasticSearch.