Connecting directly with the author of the project that inspired me is awesome.
Let's collaborate and see how we can make our two projects work together.
DuckDB has a feature that can write to SQLite: https://duckdb.org/docs/stable/core_extensions/sqlite. Starting from Parquet files, we could use DuckDB to write into SQLite databases. This could reduce ingress time to around five minutes instead of a week.
If I have some free time this weekend, I would definitely like to contribute to your project. Would you be interested?
As for my background, I focus on data engineering and data architecture. I help clients build very large-scale data pipelines, ranging from near real-time systems (under 10 ms) to large batch processing systems (handling up to 1 billion business transactions per day across thousands of partners). Some of these systems use mathematical models I developed, particularly in graph theory.
One of the things that i got interested in from the comments on my show was parquet. Everyone raving about it. Happy to see a project using that today.
Your big download plus quick refreshes is smart. Is your Background in data/AI?
Because i don’t know much about huggingface beyond its a hub for that.