Profile recomposition problem

[MatthewRalston]

This was on my mind back to this point in time. But I never had the bandwidth to fit it in amongst the other things in my life.

Needs triage and revisit scheduled.

[MatthewRalston]

@MatthewRalston needs to revisit and add items to TODO concerning recomposition and schedule, looking like back late September at this point. I'd like to revisit the assembly or the probability project.

[MatthewRalston]

Thinking, planning, and working on a Strassen multiplication algorithm and regression utilities for compositional analysis of metagenomes and genomic datasets.

Normal equations

AtAx = Atb, At(Ax - b) = 0

Strassen multiplication delegating to Numpy when n <= 4. (^_^) read eet

Want to make sure we're getting good multiplications, so no need to get too creative.

[MatthewRalston]

It would be sweet if the strassen could be finished. Implementing my own strassen doesn't add anything, but it would show these pesky recruiters what kind of ownership I take over understanding the nuts and bolts of regression and how that plays into profile decomp/recomp

[MatthewRalston]

Id still say this is a number 2 or 3 priority. In all sense of things the process should be just a regression but it reminds me of some compositional analysis algorithms like kraken and kin.

Why compositional analysis?

I'm not sure what I'm planning on doing here quite yet. Composite profiles was a breeze += but doing a strassen on CPU with Cython has been a bit of a headache. I haven't touched it since 2024 I think.

Honestly this was one of the most have features for academic use and it's long been a priority. There are other priorities right now and everything I want my life to be is taking a backseat to this gd situation. Even if I wanted to work on kmerdb full time I think I'd struggle to get sponsorship for it because there aren't that many killer features.

The most killer features is the interface at this point honestly. I'm proud of it. It does something and most people aren't really sure what that is.

Where this got tabled

Lol the assembly thing had a neat opportunity for on-disk assembly strategy. Idealistic and perfectionist, and definitely an overreach....but I'm still reaching for this part of the program to come together: sequence assembly...with scratch.

But...it came all the way from a PostgreSQL and SQLite3 database to a design for on-disk indexing into the bgzf. I had some nifty ideas that I ripped straight out of samtools. So...samtools had the nifty ideas...I just pilfered them? That's what I meant to say. Sleight of hend. 😝👋🏾

Uh...but anyways it was supposed to be a way to do on-disk assembly, that's what the .kdbg file was for. Presumably most assemblers and kmer-based community classifiers have a core issue of memory during use.

Commodity hardware?

And most students these days have the processing power to power through a calculation, but running it on commodity hardware like a 8core 16Gb low cost cloud node or container...or a laptop in a dorm room...could be the difference between finishing your thesis and leaving your laptop open all night getting your motherboard fried and blisters on your wrists.

So... The idea was to eventually index into the .bgzf file (.kdbg via .kdbi.1(,2,...etc) index files) to do on-disk based assembly. Throw multiple cores at the problem or even a GPU solution and you could save tons of time, so long as the algorithm is written to be effective at storing the bulk of the edges or kmer nodes or whatever on the SSD/HDD and only pulling up what you need via some chunk of the index file, at a time.

I don't really need to develop this feature...as most students I want to make this for can run on local cluster anyways...it's not that big a deal.

What in the... on-disk??

But for larger k and eukaryotes...and larger and larger sequencing files,... the number of total_kmers grows dramatically. K>10 and k<15 just might not be the right fit for everyone's issues. Id like to support this kind of interesting use case of bgzf, but currently the encoding I'm using is the best I can do without sinking more time into on-disk optimizations, chunking, and the opportunity cost of developing real features. Make sense?